Is there a cohomology theory for algebraic groups which captures the variety structure and restricts to the ordinary group cohomology under certain conditions.

$\begingroup$ Many thanks to Professors Humphreys and McNinch for their insightful answers. $\endgroup$– user12244Jan 16 '11 at 7:31
For a full treatment of the foundations it's best to consult Part I of the book Representations of Algebraic Groups by J.C. Jantzen (2nd ed., AMS, 2003) even though it's not easily available online. Rational (or Hochschild) cohomology has been well developed, including the broader scheme framework (DemazureGabriel book and Jantzen). What CPS and van der Kallen do in their important paper here is to relate indirectly the algebraic group cohomology with finite group cohomology for related finite groups of Lie type. This theme has been much further developed in many later papers, but is subtle.
For the algebraic groups themselves, this kind of cohomology theory has also been studied in many papers; but relating it to abstract group cohomology for the algebraic (rather than finite) groups such as the special linear group is not at all obvious.
By the way, the Inventiones paper and some others by CPS et al. are freely available online through http://gdz.sub.unigoettingen.de (just do a quick search for Parshall).
ADDED: Maybe I can answer the original question in more detail and respond to Ralph's further question. For an affine group scheme $G$ over a field $k$, rational (Hochschild) cohomology is defined as usual in terms of derived functors of the fixed point functor. But everything is done in the category of rational $G$modules; for an affine algebraic group over an algebraically closed field and finite dimensional modules this means that representing matrices have coordinate functions in $k[G]$.
Hochschild realized that for groups with added structure, one must use injective resolutions (there are usually not enough projectives). in any case, rational cohomology tends to diverge a lot from the usual group cohomology. In characteristic 0, you are essentially getting Lie algebra cohomology. Studying rational $G$modules is equivalent to studying modules for the Hopf dual of $k[G]$ (hyperalgebra, or algebra of distributions). So the answer to Ralph's question is yes: the notions of cohomology agree.
Jantzen's main focus is on prime characteristic and reductive algebraic groups, where powers of the Frobenius map yield kernels which are finite group schemes. Roughly speaking, injectives for $G$ are direct limits of injectives for finite dimensional hyperalgebras, starting with the restricted enveloping algebra of the Lie algebra of $G$ (whose cohomology usually differs from the ordinary Lie algebra cohomology). Relating rational cohomology of $G$ to ordinary cohomology of finite subgroups gets even more subtle, as discussed above. By now there is a lot of literature on these matters but many unanswered questions.

1$\begingroup$ @Jim: Let $G$ be a finite group scheme (defined over the field $k$) and let $A$ be the corresponding cocommutative hopf algebra (i.e. A is the dual hopf algebra of the coordinate ring $k[G]$ of $G$). Is there a connection between the rational cohomology of $G$ and the cohomology of $A$ defined as $Ext_A(k,)$ ? $\endgroup$– RalphJan 14 '11 at 23:27

5$\begingroup$ The first edition of Jantzen's book is available online: gen.lib.rus.ec/… $\endgroup$ Jan 15 '11 at 4:41

2$\begingroup$ @Dmitri: This can be useful for the immediate purpose, since the foundational Part I in the original 1987 Academic Press edition is essentially unchanged in the newer edition (though the longer Part II has been greatly expanded and partly rewritten). $\endgroup$ Jan 15 '11 at 13:31

$\begingroup$ Jim, thanks for this information. It follows in particular that properties of hopf algebra cohomology (like cup products, Steenrod operations, Tate cohomology, etc.) carry over to the cohomology of group schemes. $\endgroup$– RalphJan 15 '11 at 20:46

1$\begingroup$ A comment about the fifth paragraph of Jim's answer: If $G$ is a finite group scheme, then the category of rational $G$modules is equivalent to the category of $Dist(G)$modules, where $Dist(G) = k[G]^*$ is the cocommutative Hopf algebra dual to the coordinate algebra $k[G]$. But in general, if $G$ is a nonfinite affine group scheme, then rational $G$modules correspond to $Dist(G)$modules that have some local finiteness assumption (cf. Theorems 6.8 and 9.4 of the CPS paper Cohomology, Hyperalgebras, and Representations). $\endgroup$ Sep 12 '17 at 17:49
This is largely redundant with Jim Humphrey's answer, but I thought I'd add the following remarks. Ordinary group cohomology is defined via derived functors, but can be described using cocycles  this amounts to taking an explicit free resolution of the trivial module. In the setting of an algebraic group, you can also describe cohomology via cocycles; here the cocycles you should take are regular functions.
More precisely: If $G$ is a (linear) algebraic group over a field $k$, and if $V$ is a finite dimensional linear representation of $G$ as $k$algebraic group ("rational repr"), one can consider the group $C^i(G,V)$ of all regular functions $$\prod^iG=G \times \cdots \times G \to V;$$ using the "usual" boundary mappings for group cohomology, $C^\bullet(G,V)$ can be viewed as a complex. The key feature is that the cohomology of the complex $C^\bullet(G,V)$ coincides with the derived functor cohomology of $V$ in the category of rational representations of $G$. (I'm suppressing here the correct definition of $C^\bullet(G,V)$ for infinite dimensional rational representations $V$ of $G$).
This point of view makes (more?) clear how this "algebraic" cohomology can diverge from "ordinary" cohomology. Consider e.g. the additive group $G = \mathbf{G}_a$ over $k$, and consider the trivial repr. $V = k$. The algebraic cohomology $H^1(\mathbf{G}_a,k)$ identifies with the set of additive regular functions $\mathbf{G}_a \to k$; this is 1dimensional if $k$ has char. 0, while if $k$ has char. $p>0$ this cohomology has a $k$basis of the form {$T^{p^i} \mid i \ge 0$ } (for a suitable regular function $T:\mathbf{G}_a \to k$). On the other hand, the "ordinary" first cohomology for the group $k = \mathbf{G}_a(k)$ is just the set of all "abstract" group homomorphisms $k \to k$. In general, there are many such homomorphisms which are not regular functions (e.g. take $p$throots of the function $T$ in positive characteristic).
Yes, it's called "rational cohomology"  not to be confused with cohomology with rational coefficients... see eg "Rational and Generic cohomology" by Cline, Parshall, Scott and van der Kallen, Inventiones.
By using google I have even found a link, make sure it is legal for you to download this file:

1$\begingroup$ I've added the GDZ archive link to that paper in my answer. They seem to have most of the older German journals freely accessible online. $\endgroup$ Sep 22 '15 at 16:21
This post came up on my RSS feed, and I didn't see that it had been answered in 2011, but since I've now written the following, you may as well have it ...
One other point, which I think is relevant to your question. If you take $G$ over $k=\bar k$, char $k=p$ with a Frobenius map $F$ whose fixed points $G^F$ is a split Chevalley group $G^F=G(q)$ with $q=p^r$, then CPSvdK in their 1977 Inventiones paper explain a strong connection between the rational cohomology $H^i(G,M)$ and $H^i(G(q),M)$ for $M$ a finitedimensional rational $G$module. As other posters have pointed out, there is divergence between the two, even for $q$ sufficiently large. One has to perform sufficiently many Frobenius twists to $M$ to make the two sides agree. The first example is for $G=SL_2$ and $k$ has characteristic $2$. Then if the natural module is $V=L(1)$, this has $H^1(SL_2,V)=0$ in the rational setting but for all $r>1$, $H^1(SL_2(2^r),V)$ is $1$dimensional. The 'generic cohomology' is achieved as the stable limit $H^i(G(q),M)$ for $q=p^r$ as $r\to \infty$. So $H_\mathrm{gen}^1(SL_2,V)\cong k$.
What's the difference between $H^i$ and $H^i_\mathrm{gen}$? Well, with a little computation you can see that in order to get the nontrivial cocycles of $H^1_\mathrm{gen}(SL_2,V)$ in $H^1(SL_2,V)$ you would have to be able to take square roots. Well, of course, you can't do this in the category of algebraic morphisms.
But there is also another cohomology coming from forgetting all variety structure and allowing maps to be nonmorphisms. So we're just looking at $G$ as a(n infinite) group. We could denote this $H^i_{\mathrm{abs}}(G,M)$ (where the 'abs' is for abstract). In the category of abstract $G$modules, of course you can take any set theoretic map you like, so square roots are fine. And in fact one has an isomorphism $H^i(G_\mathrm{abs},M)\cong H^i(G_\mathrm{gen},M)$. I am led to believe that there is an article by Brian Parshall, [EDIT: with thanks to Wilberd van der Kallen] 'Cohomology of algebraic groups' in [The Arcata conference on Representations of Finite Groups, Proc Symp Pure Math 47 Part 1] which gives a proof of this for finitedimensional rational modules $M$ using an argument he attributes to van der Kallen.

$\begingroup$ Why refer to something that is difficult to get hold of? Just refer to our 1977 Inventiones paper which was a joint paper for good reasons. $\endgroup$ Sep 23 '15 at 7:02

$\begingroup$ @WilberdvanderKallen, becauseand I got this from the horses' mouths, if Brian and Len do not mind being compared to horsesin your 1977 paper you do not prove that abstract cohomology and generic cohomology are the same. $\endgroup$ Sep 23 '15 at 13:42

$\begingroup$ @David_Stuart. Am I missing a projective limit? Is the argument of van der Kallen that this is the kind of limit that is treated by J. E. Roos in LNM 92, Berlin 1969 ? $\endgroup$ Sep 23 '15 at 14:01

1$\begingroup$ @David_Stuart The paper by Brian is called "Cohomology of Algebraic groups" and it explains that generic cohomology of a finite dimensional module equals discrete cohomology because the projective limit satisfies the MittagLeffler condition. $\endgroup$ Sep 23 '15 at 14:48

$\begingroup$ @WilberdvanderKallen Aha. Yes. Somehow I have found that paper in my collection and it is Theorem 4(d). (And the argument is, as claimed, attributed to someone called van der Kallen.) $\endgroup$ Sep 23 '15 at 23:36