Summarize document.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Imports\n",
    "from pickle import dump, load\n",
    "import nltk\n",
    "from nltk import word_tokenize,FreqDist\n",
    "import re\n",
    "from nltk.corpus import wordnet as wn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc_list =load(open(\"data/Proxima_doc_content\",'rb'))\n",
    "len(doc_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['comment_list', 'text'])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Start working on one document and associated comments\n",
    "document = doc_list[0]\n",
    "document.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Convert bs4 ResultSet to string\n",
    "document_text = []\n",
    "for item in document['text']:\n",
    "    document_text.append(str(item))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<pre> [Federal Register Volume 79, Number 198 (Tuesday, October 14, 2014)] [Proposed Rules] [Pages 61587-61590] From the Federal Register Online via the Government Printing Office [<a href=\"http://www.gpo.gov\">www.gpo.gov</a>] [FR Doc No: 2014-24324]    ========================================================================  Proposed Rules                                                  Federal Register  ________________________________________________________________________    This section of the FEDERAL REGISTER contains notices to the public of   the proposed issuance of rules and regulations. The purpose of these   notices is to give interested persons an opportunity to participate in   the rule making prior to the adoption of the final rules.    ========================================================================      Federal Register / Vol. 79, No. 198 / Tuesday, October 14, 2014 /  Proposed Rules    [[Page 61587]]    DEPARTMENT OF THE INTERIOR  National Park Service  36 CFR Part 7  [NPS-CUVA-13806; PXXCUVA0012] RIN 1024-AE18   Special Regulations; Areas of the National Park System, Cuyahoga  Valley National Park, Bicycling  AGENCY: National Park Service, Interior.  ACTION: Proposed rule.  -----------------------------------------------------------------------  SUMMARY: Through the preparation of a Final Trail Management Plan, the  National Park Service proposed to expand recreational bicycling  opportunities in Cuyahoga Valley National Park. The Final Trail  Management Plan evaluated approximately 10 miles of new, single-track,  off-road bicycle trails in undeveloped regions of the park; 3.1 miles  of new cross-country trails; and 1.35 miles of new connector, multi-use  trails for bicycle use. The National Park Service general regulation  pertaining to bicycles requires promulgation of a special regulation to  authorize bicycle use on new trails constructed outside of developed  areas.  DATES: Comments must be received by December 15, 2014.  ADDRESSES: You may submit comments, identified by the Regulation  Identifier Number (RIN) 1024-AE18, by any of the following methods:     <bullet> Electronically: Follow the instructions for submitting  comments online at the Federal rulemaking portal: <a href=\"http://www.regulations.gov\">http://www.regulations.gov</a>.     <bullet> Hard copy: Mail or hand deliver to: Superintendent,  Cuyahoga Valley National Park, 15610 Vaughn Road, Brecksville, OH  44141.     We will only accept comments as noted above. We will not accept  comments via email, fax, or any other methods.  FOR FURTHER INFORMATION CONTACT: Chris Ryan, Chief Ranger, Cuyahoga  Valley National Park, 440-546-5940.  SUPPLEMENTARY INFORMATION:  Background  Legislation and Purposes of Cuyahoga Valley National Park      On December 27, 1974, President Gerald Ford signed Public Law 93- 555 creating Cuyahoga Valley National Recreation Area for the purpose  of ``preserving and protecting for public use and enjoyment, the  historic, scenic, natural, and recreational values of the Cuyahoga  River and the adjacent lands of the Cuyahoga Valley and for the purpose  of providing for the maintenance of needed recreational open space  necessary to the urban environment.\\'\\' In 2000, Congress redesignated  Cuyahoga Valley National Recreation Area as Cuyahoga Valley National  Park (CVNP or Park) with the passage of the Department of the Interior  and Related Agencies Appropriations Act (Pub. L. 106-291).     Cuyahoga Valley National Park is an important national resource  within a predominantly metropolitan region, where the Park is visited  by approximately 2,500,000 people annually. Located in Cuyahoga and  Summit Counties, Ohio, and situated between the cities of Cleveland and  Akron, CVNP includes approximately 30,000 acres of land, with 18,000  acres under the ownership of the National Park Service (NPS). The Park  contains significant resources, including the Cuyahoga River Valley and  its associated ecological functions, rich cultural resources and  landscapes, and a variety of recreational and outdoor use  opportunities.     In the 1930\\'s the Cuyahoga Valley provided a respite for urban  dwellers from Cleveland and Akron. During this time period, private  estates in the Cuyahoga Valley had established trails and carriage  roads for their private recreational enjoyment, including places like  the Old Carriage trail area and the Wetmore trails. Over the years,  these lands and other park lands were incorporated into the Cleveland  Metroparks and Summit Metro Parks that are now part of what is  designated as CVNP. Two significant trail corridors accelerated the  recreational connections to the Valley: the conversion of an abandoned  railroad bed to the Bike and Hike Trail in 1970 and the construction of  the Towpath Trail in the late 1980\\'s and early 1990\\'s. Many of the  trails from the earliest days of Cuyahoga Valley as a recreation  destination remain today for visitors to enjoy and share the experience  that has remained for over a century.     The Park\\'s General Management Plan/Environmental Impact Statement  (GMP) confirms the purpose, significance, and special mandates of the  Park. According to the Park\\'s GMP, one of the significant purposes of  CVNP is to ``[preserve] a landscape reminiscent of simpler times, a  place where recreation can be a gradual process of perceiving and  appreciating the roots of our contemporary existence.\\'\\' The GMP also  provides direction for park management during land acquisition and  provides a framework for NPS managers to use when making decisions  about how to conserve the Park\\'s resources and manage visitor uses in  the Park. Resource preservation for compatible recreational use is the  overall concept for management and development of the Park.  Current Status of Trails and Associated Facilities      Regional recreational trail networks have blossomed across  Northeastern Ohio, increasing demands for additional trail connections,  new trail uses, and expanded recreational opportunities. Today, the  Park contains approximately 175 miles of trails, approximately 97 miles  of which are managed by the NPS. The NPS trail system consists of three  long-distance trails--the Towpath Trail, Buckeye Trail and Valley  Bridle Trail--and eleven smaller localized trail systems with separate  access points. The park currently has one limited community connector  through the Old Carriage Trail connector trail in the northern portion  of the park and has some portions of the primary roadways improved for  bike use. Metropark partners provide five additional trail systems  within their units inside CVNP, and another trail, the Buckeye Trail,  is managed by the Buckeye Trail Association. Currently, the Park  provides access to all its trails through 25 trailheads and from the  four primary Visitor Contact Centers.     These trails provide for various uses, including 34 miles for  hiking and trail running only, 22 miles for multipurpose biking and  hiking, 17 miles for cross-country skiing, and 35 miles for equestrian  riding. Nonetheless, requests  [[Page 61588]]  for new trail uses to meet the needs of growing user groups have become  more frequent in recent years. Technologies exist today (such as  personal mobility devices) that provide new means to enjoy trails.  Walk[hyphen]in camping is a desired amenity that recently was approved  for the first time in the park. Trail running is increasing in  popularity, and biking has grown into a major recreational activity  within the Cuyahoga Valley.  2009 Comprehensive Trails Management Plan      In 1985, the Park\\'s first Trail Management Plan was developed as  the primary document to initiate many trails in the Park. The 1985  Trail Plan identified 105 miles of existing trails and proposed and  evaluated 115 miles of new trail. An additional 46 miles of trails were  identified for future consideration but were not evaluated in the 1985  Trail Plan. In 2009, CVNP initiated the development of a Comprehensive  Trails Management Plan/Environmental Impact Statement (2009 Plan/EIS)  to guide the expansion, restoration, management, operations, and use of  the trail system and its associated amenities over the next 15 years,  while keeping with the purpose, mission, and significance of the Park.  Some trails proposed in the 1985 Trail Plan but not yet implemented  were considered as part of the 2009 Plan/EIS.     The goals of the 2009 Plan/EIS were to develop a trail network  that:     <bullet> Provides experiences for a variety of trail users;     <bullet> shares the historic, scenic, natural and recreational  significance of the Park;     <bullet> minimizes impacts to the park\\'s historic, scenic, natural  and recreational resources;     <bullet> can be sustained; and     <bullet> engages cooperative partnerships that contribute to the  success of the Park\\'s trail network.     The Park conducted internal scoping with Park staff, regional park  district partners, and the Conservancy for Cuyahoga Valley National  Park and external scoping, including the mailing and distribution of  four separate newsletters, nine public meetings, and a 60-day public  comment period. As a result of this process, eight alternatives,  including a ``No Action Alternative,\\'\\' for the Park\\'s Trail Management  Plan were developed.     The 2009 Plan/EIS was completed in 2013. The Record of Decision  (ROD), signed by the NPS Midwest Regional Director on August 8, 2013,  identifies Alternative 5 as the Preferred Alternative for  implementation. Under this Alternative, approximately 14.5 miles of new  bicycle trails could be constructed in undeveloped regions of the park  and authorized by special regulations for bicycle use. The Alternative  also considers that approximately eight additional miles of existing  trail or roadways could be authorized for bicycle use in the future.  The construction and authorization of all trails for bicycle use will  be conditional on funding and the implementation of the Sustainable  Trail Guidelines, described in detail in Appendix C of the 2009 Plan/ EIS.     Due to the age and conceptual nature of the 1977 GMP, a 2013  Foundation Plan was developed for the Park that identifies active  recreation and implementation of the 2009 Plan/EIS as an objective to  meets its goals. The 2009 Plan/EIS and ROD may be viewed online at  <a href=\"http://parkplanning.nps.gov/cuyahogatrailplan\">http://parkplanning.nps.gov/cuyahogatrailplan</a>.  Construction and Management of the Bicycle Trails      Many of the proposed bicycle trails have not yet been built and  would not be immediately open for use. An Implementation Strategy Plan  is under development to prioritize trail projects and assemble the  additional planning, funding, staffing, project management, and  monitoring that will be needed to accomplish them successfully. The  Trails Forever Program, administered by the Park in partnership with  the Conservancy for Cuyahoga Valley National Park, will be a critical  component in the development, implementation, and sustainability of the  trails in the Park. The Trails Forever Program will advance its role in  the coming years as a primary component for Trail Plan funding and will  work with Park Staff to annually prioritize trail projects. As part of  the initial implementation phase of the Trail Plan, the Park will  evaluate the Trails Forever program for staffing, program coordination,  and program direction to ensure its future success and its alignment  with the goals of the Trail Plan.     Volunteers for trail work at the Park will continue to be a vital  component of trail stewardship in the Park. Management and coordination  of volunteers will continue through the joint Volunteer Program office  of the Park and the Conservancy for Cuyahoga Valley National Park. The  use of Park staff and the existing volunteer trail groups to monitor  and mitigate the environmental impacts of bicycle use on these trails  will ensure that the trails are maintained in good condition and that  any issues of concern are immediately brought to the attention of Park  management. In addition, the Park will continue to update its Sign Plan  and upgrade park and trail signs accordingly. As trail signs are  updated, trail accessibility information for each trail will be made  available to the public.  Proposed Rule      The proposed rule complies with the general requirement of 36 CFR  4.30, which requires a special regulation to designate new bicycle  routes off park roads and outside of developed areas. To provide users  with additional recreational bicycling opportunities, this proposed  rule would authorize the Superintendent to designate any of the  following trails as routes for bicycle use:  ----------------------------------------------------------------------------------------------------------------                                     Approximate            Trail name                  length          Surface type        Usage type           Description ---------------------------------------------------------------------------------------------------------------- East Rim.......................  10 miles.........  Natural surface..  Off-road, single-  Approximately ten                                                                         track bicycle.     miles of a loop                                                                                            system trail of                                                                                            varying distances                                                                                            along the east                                                                                            central portion of                                                                                            the Park, north of                                                                                            Old Akron-Peninsula                                                                                            Road and south of                                                                                            Brandywine Falls                                                                                            trailhead, near the                                                                                            Krecjic Restoration                                                                                            Site. High Meadow Trail..............  3.1 miles........  Natural surface..  Cross-country....  Located west of Blue                                                                                            Hen Falls, near the                                                                                            existing Kurowski                                                                                            fields and linking to                                                                                            the existing Buckeye                                                                                            Trail. Old Carriage Connector Trail...  0.35 miles.......  Crushed gravel...  Multi-purpose....  Extension of existing                                                                                            Old Carriage Road                                                                                            connector to existing                                                                                            Bike and Hike Trail.  [[Page 61589]]    Highland Connector Trail.......  1.0 miles........  Crushed gravel...  Multi-purpose....  New connector from                                                                                            existing Bike and                                                                                            Hike Trail to                                                                                            existing Towpath                                                                                            Trail on south side                                                                                            of Highland Road,                                                                                            extending on the                                                                                            north side of                                                                                            Highland Road from                                                                                            Towpath to the Vaughn                                                                                            overflow parking                                                                                            area. ----------------------------------------------------------------------------------------------------------------      After trail construction is completed, but before a trail is  designated for bicycle use, the Superintendent would be required to  issue a written determination that the route is open for public use and  that such bicycle use is consistent with the 2013 park plan for bicycle  use, including implementation of the park\\'s sustainable trail  guidelines with monitoring and mitigation through adaptive management.  This would ensure that bicycle use remains consistent with the  protection of the park area\\'s natural, scenic and aesthetic values,  safety considerations and management objectives, and will not disturb  wildlife or park resources. No additional NEPA compliance would be  necessary beyond the 2013 EIS/ROD, and the written determination would  be added into the park\\'s administrative file for the trail project. The  Superintendent would provide notice of such designation through one or  more of the public notice procedures outlined in 36 CFR 1.7.     The proposed rule would also authorize the Superintendent to impose  closures or restrictions for bicycle use on designated routes after  taking into consideration public health and safety, resource  protection, and other management activities and objectives, provided  public notice is given under 36 CFR 1.7.  Compliance With Other Laws, Executive Orders, and Department Policy  Regulatory Planning and Review (Executive Orders 12866 and 13563)      Executive Order 12866 provides that the Office of Information and  Regulatory Affairs in the Office of Management and Budget will review  all significant rules. The Office of Information and Regulatory Affairs  has determined that this rule is not significant.     Executive Order 13563 reaffirms the principles of E.O. 12866 while  calling for improvements in the nation\\'s regulatory system to promote  predictability, to reduce uncertainty, and to use the best, most  innovative, and least burdensome tools for achieving regulatory ends.  The executive order directs agencies to consider regulatory approaches  that reduce burdens and maintain flexibility and freedom of choice for  the public where these approaches are relevant, feasible, and  consistent with regulatory objectives. E.O. 13563 emphasizes further  that regulations must be based on the best available science and that  the rulemaking process must allow for public participation and an open  exchange of ideas. We have developed this rule in a manner consistent  with these requirements.  Regulatory Flexibility Act (RFA)      This rule will not have a significant economic effect on a  substantial number of small entities under the RFA (5 U.S.C. 601 et  seq.). This certification is based on information contained in the  report titled, ``Cost-Benefit and Regulatory Flexibility Analyses:  Proposed Regulations to Designate Bicycle Routes in Cuyahoga Valley  National Park\\'\\' that is available for review at <a href=\"http://parkplanning.nps.gov/cuyahogatrailplan\">http://parkplanning.nps.gov/cuyahogatrailplan</a>.  Small Business Regulatory Enforcement Fairness Act (SBREFA)      This rule is not a major rule under 5 U.S.C. 804(2), the SBREFA.  This rule:     a. Does not have an annual effect on the economy of $100 million or  more.     b. Would not cause a major increase in costs or prices for  consumers, individual industries, Federal, State, or local government  agencies, or geographic regions.     c. Does not have significant adverse effects on competition,  employment, investment, productivity, innovation, or the ability of  U.S.-based enterprises to compete with foreign-based enterprises.     The current and anticipated users of bicycle routes in CUVA are  predominantly individuals engaged in recreational activities. There are  no businesses in the surrounding area that would be adversely affected  by bicycle use of these trails. Although the park does not have any  bicycle rental concessioners, there is a bicycle rental facility  adjacent to the park that provides bike rentals that are used within  CUVA.  Unfunded Mandates Reform Act (UMRA)      This rule does not impose an unfunded mandate on State, local, or  tribal governments or the private sector of more than $100 million per  year. The rule does not have a significant or unique effect on State,  local or tribal governments or the private sector. A statement  containing the information required by the UMRA (2 U.S.C. 1531 et seq.)  is not required.  Takings (Executive Order 12630)      This rule does not affect a taking of private property or otherwise  have taking implications under Executive Order 12630. A taking  implications assessment is not required because this rule will not deny  any private property owner of beneficial uses of their land, nor will  it significantly reduce their land\\'s value.  Federalism (Executive Order 13132)      Under the criteria in section 1 of Executive Order 13132, this rule  does not have sufficient federalism implications to warrant the  preparation of a Federalism summary impact statement. A Federalism  summary impact statement is not required.  Civil Justice Reform (Executive Order 12988)      This rule complies with the requirements of Executive Order 12988.  Specifically this rule:     (a) Meets the in the criteria of section 3(a) requiring that all  regulations be reviewed to eliminate errors and ambiguity and be  written to minimize litigation; and     (b) meets the criteria of section 3(b)(2) requiring that all  regulations be written in clear language and contain clear legal  standards.  Consultation With Indian Tribes (Executive Order 13175 and Department  Policy)      The Department of the Interior strives to strengthen its  government-to-government relationship with Indian Tribes through a  commitment to consultation with Indian Tribes and recognition of their  right to self-governance and tribal sovereignty. We have evaluated this  rule under the Department\\'s consultation policy and under the criteria  in Executive Order 13175 and have determined that it has no substantial  direct effects on federally recognized Indian tribes and that  consultation under the Department\\'s  [[Page 61590]]  tribal consultation policy is not required. Affiliated Native American  tribes were contacted by letters sent in June, 2012 and May, 2013 to  solicit any interests or concerns with the proposed action. No  responses were received by the Park.  Paperwork Reduction Act (PRA)      This rule does not contain information collection requirements, and  a submission to the Office of Management and Budget under the PRA is  not required.  National Environmental Policy Act (NEPA)      We have prepared an environmental impact statement and have  determined that this rule will not have a significant effect on the  quality of the human environment under the NEPA of 1969. The Plan/EIS  for the Park and ROD that included an evaluation of bicycling within  the proposed areas may be viewed online at <a href=\"http://parkplanning.nps.gov/cuyahogatrailplan\">http://parkplanning.nps.gov/cuyahogatrailplan</a>.  Effects on the Energy Supply (Executive Order 13211)      This rule is not a significant energy action under the definition  in Executive Order 13211. A statement of Energy Effects is not  required.  Clarity of This Regulation      We are required by Executive Orders 12866 and 12988 and by the  Presidential Memorandum of June 1, 1998, to write all rules in plain  language. This means that each rule we publish must:     (a) Be logically organized;     (b) Use the active voice to address readers directly;     (c) Use clear language rather than jargon;     (d) Be divided into short sections and sentences; and     (e) Use lists and tables wherever possible.     If you feel that we have not met these requirements, send us  comments by one of the methods listed in the ADDRESSES section. To  better help us revise the rule, your comments should be as specific as  possible. For example, you should tell us the numbers of the sections  or paragraphs that you find unclear, which sections or sentences are  too long, the sections where you feel lists or tables would be useful,  etc.  Drafting Information      The primary authors of this regulation are Lynn Garrity, Outdoor  Recreation Planner, Cuyahoga Valley National Park, and C. Rose  Wilkinson and A.J. North, NPS Regulations Program, Washington, DC.  Public Participation      It is the policy of NPS, whenever practicable, to afford the public  an opportunity to participate in the rulemaking process. Accordingly,  interested parties may submit written comments, suggestions, or  objections regarding this proposed rule only as noted at the beginning  of this rule. We will not accept comments by fax, email, or any other  methods.  Public Availability of Comments      Before including your address, phone number, email address, or  other personal identifying information in your comments, you should be  aware that your entire comment--including your personal identifying  information--may be made publicly available at any time. While you can  ask us in your comment to withhold your personal identifying  information from public review, we cannot guarantee that we will be  able to do so.  List of Subjects in 36 CFR Part 7      National Parks, Reporting and recordkeeping requirements.      In consideration of the foregoing, the NPS proposes to amend 36 CFR  part 7 as set forth below:  PART 7--SPECIAL REGULATIONS, AREAS OF THE NATIONAL PARK SYSTEM  0 1. The authority for Part 7 continues to read as follows:      Authority: 16 U.S.C. 1, 3, 9a, 462(k); Sec. 7.96 also issued  under 36 U.S.C. 501-511, D.C. Code 10-137 (2001) and D.C. Code 50- 2201 (2001).  0 2. Amend Sec.  7.17 to revise the section heading and add paragraph (b)  to read as follows:   Sec.  7.17  Cuyahoga Valley National Park.      (a) * * *     (b) Bicycles. (1) The Superintendent may designate routes or  portions of routes for bicycle use on the following trails:     (i) East Rim (approximately 10 miles);     (ii) High Meadow Trail (approximately 3.1 miles);     (iii) Old Carriage Connector Trail (approximately 0.35 miles); and     (iv) Highland Connector Trail (approximately 1.0 mile).     (2) After trail construction is complete:     (i) To designate a bicycle route, the Superintendent must make a  written determination that:     (A) The route is open for public use; and     (B) Bicycle use is consistent with the protection of the park  area\\'s natural, scenic and aesthetic values, safety considerations, and  management objectives, and will not disturb wildlife or park resources.     (ii) The Superintendent will provide public notice of all such  actions through one or more of the methods listed in Sec.  1.7 of this  chapter.     (3) The Superintendent may open or close designated routes, or  portions thereof, or impose conditions or restrictions for bicycle use  after taking into consideration public health and safety, natural and  cultural resource protection, and other management activities and  objectives.     (i) The Superintendent will provide public notice of all such  actions through one or more of the methods listed in Sec.  1.7 of this  chapter.     (ii) Violating a closure, condition, or restriction is prohibited. * * * * *      Dated: September 26, 2014. Michael Bean, Principal Deputy Assistant Secretary for Fish and Wildlife and Parks. [FR Doc. 2014-24324 Filed 10-10-14; 8:45 am] BILLING CODE 4310-EJ-P   </bullet></bullet></bullet></bullet></bullet></bullet></bullet></pre>'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = document_text[0].replace('\\n',' ')\n",
    "text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "summary_idx = text.find(\"SUMMARY:\")\n",
    "dates_idx = text.find(\"DATES:\")\n",
    "supp_info_idx = text.find(\"SUPPLEMENTARY INFORMATION:\")\n",
    "summary = text[summary_idx:dates_idx]\n",
    "suppl_info = text[supp_info_idx:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Get rid of HTML tags\n",
    "def cleanhtml(raw_html):\n",
    "    cleanr = re.compile('<.*?>')\n",
    "    cleantext = re.sub(cleanr, '', raw_html)\n",
    "    return cleantext"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "clean_suppl_info = cleanhtml(suppl_info)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"SUPPLEMENTARY INFORMATION:  Background  Legislation and Purposes of Cuyahoga Valley National Park      On December 27, 1974, President Gerald Ford signed Public Law 93- 555 creating Cuyahoga Valley National Recreation Area for the purpose  of ``preserving and protecting for public use and enjoyment, the  historic, scenic, natural, and recreational values of the Cuyahoga  River and the adjacent lands of the Cuyahoga Valley and for the purpose  of providing for the maintenance of needed recreational open space  necessary to the urban environment.'' In 2000, Congress redesignated  Cuyahoga Valley National Recreation Area as Cuyahoga Valley National  Park (CVNP or Park) with the passage of the Department of the Interior  and Related Agencies Appropriations Act (Pub. L. 106-291).     Cuyahoga Valley National Park is an important national resource  within a predominantly metropolitan region, where the Park is visited  by approximately 2,500,000 people annually. Located in Cuyahoga and  Summit Counties, Ohio, and situated between the cities of Cleveland and  Akron, CVNP includes approximately 30,000 acres of land, with 18,000  acres under the ownership of the National Park Service (NPS). The Park  contains significant resources, including the Cuyahoga River Valley and  its associated ecological functions, rich cultural resources and  landscapes, and a variety of recreational and outdoor use  opportunities.     In the 1930's the Cuyahoga Valley provided a respite for urban  dwellers from Cleveland and Akron. During this time period, private  estates in the Cuyahoga Valley had established trails and carriage  roads for their private recreational enjoyment, including places like  the Old Carriage trail area and the Wetmore trails. Over the years,  these lands and other park lands were incorporated into the Cleveland  Metroparks and Summit Metro Parks that are now part of what is  designated as CVNP. Two significant trail corridors accelerated the  recreational connections to the Valley: the conversion of an abandoned  railroad bed to the Bike and Hike Trail in 1970 and the construction of  the Towpath Trail in the late 1980's and early 1990's. Many of the  trails from the earliest days of Cuyahoga Valley as a recreation  destination remain today for visitors to enjoy and share the experience  that has remained for over a century.     The Park's General Management Plan/Environmental Impact Statement  (GMP) confirms the purpose, significance, and special mandates of the  Park. According to the Park's GMP, one of the significant purposes of  CVNP is to ``[preserve] a landscape reminiscent of simpler times, a  place where recreation can be a gradual process of perceiving and  appreciating the roots of our contemporary existence.'' The GMP also  provides direction for park management during land acquisition and  provides a framework for NPS managers to use when making decisions  about how to conserve the Park's resources and manage visitor uses in  the Park. Resource preservation for compatible recreational use is the  overall concept for management and development of the Park.  Current Status of Trails and Associated Facilities      Regional recreational trail networks have blossomed across  Northeastern Ohio, increasing demands for additional trail connections,  new trail uses, and expanded recreational opportunities. Today, the  Park contains approximately 175 miles of trails, approximately 97 miles  of which are managed by the NPS. The NPS trail system consists of three  long-distance trails--the Towpath Trail, Buckeye Trail and Valley  Bridle Trail--and eleven smaller localized trail systems with separate  access points. The park currently has one limited community connector  through the Old Carriage Trail connector trail in the northern portion  of the park and has some portions of the primary roadways improved for  bike use. Metropark partners provide five additional trail systems  within their units inside CVNP, and another trail, the Buckeye Trail,  is managed by the Buckeye Trail Association. Currently, the Park  provides access to all its trails through 25 trailheads and from the  four primary Visitor Contact Centers.     These trails provide for various uses, including 34 miles for  hiking and trail running only, 22 miles for multipurpose biking and  hiking, 17 miles for cross-country skiing, and 35 miles for equestrian  riding. Nonetheless, requests  [[Page 61588]]  for new trail uses to meet the needs of growing user groups have become  more frequent in recent years. Technologies exist today (such as  personal mobility devices) that provide new means to enjoy trails.  Walk[hyphen]in camping is a desired amenity that recently was approved  for the first time in the park. Trail running is increasing in  popularity, and biking has grown into a major recreational activity  within the Cuyahoga Valley.  2009 Comprehensive Trails Management Plan      In 1985, the Park's first Trail Management Plan was developed as  the primary document to initiate many trails in the Park. The 1985  Trail Plan identified 105 miles of existing trails and proposed and  evaluated 115 miles of new trail. An additional 46 miles of trails were  identified for future consideration but were not evaluated in the 1985  Trail Plan. In 2009, CVNP initiated the development of a Comprehensive  Trails Management Plan/Environmental Impact Statement (2009 Plan/EIS)  to guide the expansion, restoration, management, operations, and use of  the trail system and its associated amenities over the next 15 years,  while keeping with the purpose, mission, and significance of the Park.  Some trails proposed in the 1985 Trail Plan but not yet implemented  were considered as part of the 2009 Plan/EIS.     The goals of the 2009 Plan/EIS were to develop a trail network  that:      Provides experiences for a variety of trail users;      shares the historic, scenic, natural and recreational  significance of the Park;      minimizes impacts to the park's historic, scenic, natural  and recreational resources;      can be sustained; and      engages cooperative partnerships that contribute to the  success of the Park's trail network.     The Park conducted internal scoping with Park staff, regional park  district partners, and the Conservancy for Cuyahoga Valley National  Park and external scoping, including the mailing and distribution of  four separate newsletters, nine public meetings, and a 60-day public  comment period. As a result of this process, eight alternatives,  including a ``No Action Alternative,'' for the Park's Trail Management  Plan were developed.     The 2009 Plan/EIS was completed in 2013. The Record of Decision  (ROD), signed by the NPS Midwest Regional Director on August 8, 2013,  identifies Alternative 5 as the Preferred Alternative for  implementation. Under this Alternative, approximately 14.5 miles of new  bicycle trails could be constructed in undeveloped regions of the park  and authorized by special regulations for bicycle use. The Alternative  also considers that approximately eight additional miles of existing  trail or roadways could be authorized for bicycle use in the future.  The construction and authorization of all trails for bicycle use will  be conditional on funding and the implementation of the Sustainable  Trail Guidelines, described in detail in Appendix C of the 2009 Plan/ EIS.     Due to the age and conceptual nature of the 1977 GMP, a 2013  Foundation Plan was developed for the Park that identifies active  recreation and implementation of the 2009 Plan/EIS as an objective to  meets its goals. The 2009 Plan/EIS and ROD may be viewed online at  http://parkplanning.nps.gov/cuyahogatrailplan.  Construction and Management of the Bicycle Trails      Many of the proposed bicycle trails have not yet been built and  would not be immediately open for use. An Implementation Strategy Plan  is under development to prioritize trail projects and assemble the  additional planning, funding, staffing, project management, and  monitoring that will be needed to accomplish them successfully. The  Trails Forever Program, administered by the Park in partnership with  the Conservancy for Cuyahoga Valley National Park, will be a critical  component in the development, implementation, and sustainability of the  trails in the Park. The Trails Forever Program will advance its role in  the coming years as a primary component for Trail Plan funding and will  work with Park Staff to annually prioritize trail projects. As part of  the initial implementation phase of the Trail Plan, the Park will  evaluate the Trails Forever program for staffing, program coordination,  and program direction to ensure its future success and its alignment  with the goals of the Trail Plan.     Volunteers for trail work at the Park will continue to be a vital  component of trail stewardship in the Park. Management and coordination  of volunteers will continue through the joint Volunteer Program office  of the Park and the Conservancy for Cuyahoga Valley National Park. The  use of Park staff and the existing volunteer trail groups to monitor  and mitigate the environmental impacts of bicycle use on these trails  will ensure that the trails are maintained in good condition and that  any issues of concern are immediately brought to the attention of Park  management. In addition, the Park will continue to update its Sign Plan  and upgrade park and trail signs accordingly. As trail signs are  updated, trail accessibility information for each trail will be made  available to the public.  Proposed Rule      The proposed rule complies with the general requirement of 36 CFR  4.30, which requires a special regulation to designate new bicycle  routes off park roads and outside of developed areas. To provide users  with additional recreational bicycling opportunities, this proposed  rule would authorize the Superintendent to designate any of the  following trails as routes for bicycle use:  ----------------------------------------------------------------------------------------------------------------                                     Approximate            Trail name                  length          Surface type        Usage type           Description ---------------------------------------------------------------------------------------------------------------- East Rim.......................  10 miles.........  Natural surface..  Off-road, single-  Approximately ten                                                                         track bicycle.     miles of a loop                                                                                            system trail of                                                                                            varying distances                                                                                            along the east                                                                                            central portion of                                                                                            the Park, north of                                                                                            Old Akron-Peninsula                                                                                            Road and south of                                                                                            Brandywine Falls                                                                                            trailhead, near the                                                                                            Krecjic Restoration                                                                                            Site. High Meadow Trail..............  3.1 miles........  Natural surface..  Cross-country....  Located west of Blue                                                                                            Hen Falls, near the                                                                                            existing Kurowski                                                                                            fields and linking to                                                                                            the existing Buckeye                                                                                            Trail. Old Carriage Connector Trail...  0.35 miles.......  Crushed gravel...  Multi-purpose....  Extension of existing                                                                                            Old Carriage Road                                                                                            connector to existing                                                                                            Bike and Hike Trail.  [[Page 61589]]    Highland Connector Trail.......  1.0 miles........  Crushed gravel...  Multi-purpose....  New connector from                                                                                            existing Bike and                                                                                            Hike Trail to                                                                                            existing Towpath                                                                                            Trail on south side                                                                                            of Highland Road,                                                                                            extending on the                                                                                            north side of                                                                                            Highland Road from                                                                                            Towpath to the Vaughn                                                                                            overflow parking                                                                                            area. ----------------------------------------------------------------------------------------------------------------      After trail construction is completed, but before a trail is  designated for bicycle use, the Superintendent would be required to  issue a written determination that the route is open for public use and  that such bicycle use is consistent with the 2013 park plan for bicycle  use, including implementation of the park's sustainable trail  guidelines with monitoring and mitigation through adaptive management.  This would ensure that bicycle use remains consistent with the  protection of the park area's natural, scenic and aesthetic values,  safety considerations and management objectives, and will not disturb  wildlife or park resources. No additional NEPA compliance would be  necessary beyond the 2013 EIS/ROD, and the written determination would  be added into the park's administrative file for the trail project. The  Superintendent would provide notice of such designation through one or  more of the public notice procedures outlined in 36 CFR 1.7.     The proposed rule would also authorize the Superintendent to impose  closures or restrictions for bicycle use on designated routes after  taking into consideration public health and safety, resource  protection, and other management activities and objectives, provided  public notice is given under 36 CFR 1.7.  Compliance With Other Laws, Executive Orders, and Department Policy  Regulatory Planning and Review (Executive Orders 12866 and 13563)      Executive Order 12866 provides that the Office of Information and  Regulatory Affairs in the Office of Management and Budget will review  all significant rules. The Office of Information and Regulatory Affairs  has determined that this rule is not significant.     Executive Order 13563 reaffirms the principles of E.O. 12866 while  calling for improvements in the nation's regulatory system to promote  predictability, to reduce uncertainty, and to use the best, most  innovative, and least burdensome tools for achieving regulatory ends.  The executive order directs agencies to consider regulatory approaches  that reduce burdens and maintain flexibility and freedom of choice for  the public where these approaches are relevant, feasible, and  consistent with regulatory objectives. E.O. 13563 emphasizes further  that regulations must be based on the best available science and that  the rulemaking process must allow for public participation and an open  exchange of ideas. We have developed this rule in a manner consistent  with these requirements.  Regulatory Flexibility Act (RFA)      This rule will not have a significant economic effect on a  substantial number of small entities under the RFA (5 U.S.C. 601 et  seq.). This certification is based on information contained in the  report titled, ``Cost-Benefit and Regulatory Flexibility Analyses:  Proposed Regulations to Designate Bicycle Routes in Cuyahoga Valley  National Park'' that is available for review at http://parkplanning.nps.gov/cuyahogatrailplan.  Small Business Regulatory Enforcement Fairness Act (SBREFA)      This rule is not a major rule under 5 U.S.C. 804(2), the SBREFA.  This rule:     a. Does not have an annual effect on the economy of $100 million or  more.     b. Would not cause a major increase in costs or prices for  consumers, individual industries, Federal, State, or local government  agencies, or geographic regions.     c. Does not have significant adverse effects on competition,  employment, investment, productivity, innovation, or the ability of  U.S.-based enterprises to compete with foreign-based enterprises.     The current and anticipated users of bicycle routes in CUVA are  predominantly individuals engaged in recreational activities. There are  no businesses in the surrounding area that would be adversely affected  by bicycle use of these trails. Although the park does not have any  bicycle rental concessioners, there is a bicycle rental facility  adjacent to the park that provides bike rentals that are used within  CUVA.  Unfunded Mandates Reform Act (UMRA)      This rule does not impose an unfunded mandate on State, local, or  tribal governments or the private sector of more than $100 million per  year. The rule does not have a significant or unique effect on State,  local or tribal governments or the private sector. A statement  containing the information required by the UMRA (2 U.S.C. 1531 et seq.)  is not required.  Takings (Executive Order 12630)      This rule does not affect a taking of private property or otherwise  have taking implications under Executive Order 12630. A taking  implications assessment is not required because this rule will not deny  any private property owner of beneficial uses of their land, nor will  it significantly reduce their land's value.  Federalism (Executive Order 13132)      Under the criteria in section 1 of Executive Order 13132, this rule  does not have sufficient federalism implications to warrant the  preparation of a Federalism summary impact statement. A Federalism  summary impact statement is not required.  Civil Justice Reform (Executive Order 12988)      This rule complies with the requirements of Executive Order 12988.  Specifically this rule:     (a) Meets the in the criteria of section 3(a) requiring that all  regulations be reviewed to eliminate errors and ambiguity and be  written to minimize litigation; and     (b) meets the criteria of section 3(b)(2) requiring that all  regulations be written in clear language and contain clear legal  standards.  Consultation With Indian Tribes (Executive Order 13175 and Department  Policy)      The Department of the Interior strives to strengthen its  government-to-government relationship with Indian Tribes through a  commitment to consultation with Indian Tribes and recognition of their  right to self-governance and tribal sovereignty. We have evaluated this  rule under the Department's consultation policy and under the criteria  in Executive Order 13175 and have determined that it has no substantial  direct effects on federally recognized Indian tribes and that  consultation under the Department's  [[Page 61590]]  tribal consultation policy is not required. Affiliated Native American  tribes were contacted by letters sent in June, 2012 and May, 2013 to  solicit any interests or concerns with the proposed action. No  responses were received by the Park.  Paperwork Reduction Act (PRA)      This rule does not contain information collection requirements, and  a submission to the Office of Management and Budget under the PRA is  not required.  National Environmental Policy Act (NEPA)      We have prepared an environmental impact statement and have  determined that this rule will not have a significant effect on the  quality of the human environment under the NEPA of 1969. The Plan/EIS  for the Park and ROD that included an evaluation of bicycling within  the proposed areas may be viewed online at http://parkplanning.nps.gov/cuyahogatrailplan.  Effects on the Energy Supply (Executive Order 13211)      This rule is not a significant energy action under the definition  in Executive Order 13211. A statement of Energy Effects is not  required.  Clarity of This Regulation      We are required by Executive Orders 12866 and 12988 and by the  Presidential Memorandum of June 1, 1998, to write all rules in plain  language. This means that each rule we publish must:     (a) Be logically organized;     (b) Use the active voice to address readers directly;     (c) Use clear language rather than jargon;     (d) Be divided into short sections and sentences; and     (e) Use lists and tables wherever possible.     If you feel that we have not met these requirements, send us  comments by one of the methods listed in the ADDRESSES section. To  better help us revise the rule, your comments should be as specific as  possible. For example, you should tell us the numbers of the sections  or paragraphs that you find unclear, which sections or sentences are  too long, the sections where you feel lists or tables would be useful,  etc.  Drafting Information      The primary authors of this regulation are Lynn Garrity, Outdoor  Recreation Planner, Cuyahoga Valley National Park, and C. Rose  Wilkinson and A.J. North, NPS Regulations Program, Washington, DC.  Public Participation      It is the policy of NPS, whenever practicable, to afford the public  an opportunity to participate in the rulemaking process. Accordingly,  interested parties may submit written comments, suggestions, or  objections regarding this proposed rule only as noted at the beginning  of this rule. We will not accept comments by fax, email, or any other  methods.  Public Availability of Comments      Before including your address, phone number, email address, or  other personal identifying information in your comments, you should be  aware that your entire comment--including your personal identifying  information--may be made publicly available at any time. While you can  ask us in your comment to withhold your personal identifying  information from public review, we cannot guarantee that we will be  able to do so.  List of Subjects in 36 CFR Part 7      National Parks, Reporting and recordkeeping requirements.      In consideration of the foregoing, the NPS proposes to amend 36 CFR  part 7 as set forth below:  PART 7--SPECIAL REGULATIONS, AREAS OF THE NATIONAL PARK SYSTEM  0 1. The authority for Part 7 continues to read as follows:      Authority: 16 U.S.C. 1, 3, 9a, 462(k); Sec. 7.96 also issued  under 36 U.S.C. 501-511, D.C. Code 10-137 (2001) and D.C. Code 50- 2201 (2001).  0 2. Amend Sec.  7.17 to revise the section heading and add paragraph (b)  to read as follows:   Sec.  7.17  Cuyahoga Valley National Park.      (a) * * *     (b) Bicycles. (1) The Superintendent may designate routes or  portions of routes for bicycle use on the following trails:     (i) East Rim (approximately 10 miles);     (ii) High Meadow Trail (approximately 3.1 miles);     (iii) Old Carriage Connector Trail (approximately 0.35 miles); and     (iv) Highland Connector Trail (approximately 1.0 mile).     (2) After trail construction is complete:     (i) To designate a bicycle route, the Superintendent must make a  written determination that:     (A) The route is open for public use; and     (B) Bicycle use is consistent with the protection of the park  area's natural, scenic and aesthetic values, safety considerations, and  management objectives, and will not disturb wildlife or park resources.     (ii) The Superintendent will provide public notice of all such  actions through one or more of the methods listed in Sec.  1.7 of this  chapter.     (3) The Superintendent may open or close designated routes, or  portions thereof, or impose conditions or restrictions for bicycle use  after taking into consideration public health and safety, natural and  cultural resource protection, and other management activities and  objectives.     (i) The Superintendent will provide public notice of all such  actions through one or more of the methods listed in Sec.  1.7 of this  chapter.     (ii) Violating a closure, condition, or restriction is prohibited. * * * * *      Dated: September 26, 2014. Michael Bean, Principal Deputy Assistant Secretary for Fish and Wildlife and Parks. [FR Doc. 2014-24324 Filed 10-10-14; 8:45 am] BILLING CODE 4310-EJ-P   \""
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_suppl_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Now extract keywords from supplement information **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def tokenize_text(corpus):\n",
    "    pattern = r'''(?x)    # set flag to allow verbose regexps\n",
    "    (([A-Z]\\.)+)       # abbreviations, e.g. B.C.\n",
    "    |(\\w+([-']\\w+)*)       # words with optional internal hyphens e.g. after-ages or author's\n",
    "    '''\n",
    "    tokens = nltk.regexp_tokenize(corpus,pattern)\n",
    "    all_token = [word.lower() for token in tokens for word in token if word != \"\" and word[0] != \"'\" and word[0] != \"-\"]\n",
    "    return all_token\n",
    "\n",
    "def tokenize_text_sent(corpus):\n",
    "    sent_tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')\n",
    "    raw_sents = sent_tokenizer.tokenize(corpus) # Split text into sentences    \n",
    "    return [tokenize_text(sent) for sent in raw_sents]\n",
    "\n",
    "def tag_my_text(sents):\n",
    "    return [nltk.pos_tag(sent) for sent in sents]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "suppl_info_tokens = tokenize_text(clean_suppl_info)\n",
    "suppl_info_sent_tokens = tokenize_text_sent(clean_suppl_info)\n",
    "pos_tag_text = tag_my_text(suppl_info_sent_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# matchobj = re.findall(r'(January|February|March|April|May|June|July|August|September|October|November|December)\n",
    "#(\\s\\d{2}\\,\\s\\d{4})',clean_suppl_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "144 sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Chunk noun phrases in tree \n",
    "def noun_phrase_chunker():\n",
    "    grammar = r\"\"\"\n",
    "    NP: {<DT|PP\\$>?<JJ>*<NN>}   # chunk determiner/possessive, adjectives and noun\n",
    "    \"\"\"\n",
    "    cp = nltk.RegexpParser(grammar)\n",
    "    return cp\n",
    "\n",
    "#Extract only the NP marked phrases from the parse tree, that is the chunk we defined\n",
    "def noun_phrase_extractor(sentences, chunker):\n",
    "    res = []\n",
    "    for sent in sentences:\n",
    "        tree = chunker.parse(sent)\n",
    "        for subtree in tree.subtrees():\n",
    "            if subtree.label() == 'NP' : \n",
    "                res.append(subtree[0:len(subtree)])\n",
    "                #res.append(subtree[0])\n",
    "                #print(subtree)\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#remove tags and get only the noun phrases , can be adjusted for length\n",
    "def noun_phrase_finder(tagged_text):\n",
    "    all_proper_noun = noun_phrase_extractor(tagged_text,noun_phrase_chunker()) \n",
    "    #does not literally mean proper noun. Chunker only extracts common noun\n",
    "    noun_phrase_list = []                                                      \n",
    "    #noun_phrase_string_list =[]\n",
    "    for noun_phrase in all_proper_noun:\n",
    "        if len(noun_phrase) > 0: #this means where the size of the phrase is greater than 1\n",
    "            small_list =[]\n",
    "            for (word,tag) in noun_phrase:\n",
    "                small_list.append(word)\n",
    "            noun_phrase_list.append(small_list)\n",
    "            #noun_phrase_string_list.append(' '.join(small_list))\n",
    "    return noun_phrase_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np_list = noun_phrase_finder(pos_tag_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#get frequency dist of different length in all the noun phrases extracted. \n",
    "#Something of the form {1:45,2:23} - how many 1phrased and 2 phrased chunks I have etc.\n",
    "def get_length_np(nounPhrase):\n",
    "    np_length={}\n",
    "    for inner_np in nounPhrase:\n",
    "        np_length[len(inner_np)] = np_length.get(len(inner_np),0) + 1\n",
    "    return np_length\n",
    "\n",
    "#get freq dist obj for noun phrase of different lengths\n",
    "def find_freq(nested_list,nest_len):\n",
    "    #from nltk.probability import FreqDist\n",
    "    fdist_list =[]\n",
    "    for inner_np in nested_list:\n",
    "        if len(inner_np) == nest_len:\n",
    "            fdist_list.append(' '.join(inner_np))\n",
    "    fdist = FreqDist(fdist_list)\n",
    "    return fdist\n",
    "\n",
    "#make a grand list of top occuring noun phrases of different sizes\n",
    "def get_top_np(np):\n",
    "    master_common_list=[]\n",
    "    len_list =get_length_np(np).keys()\n",
    "    for item in len_list:\n",
    "        fdist_np = find_freq(np_list,item)\n",
    "        master_common_list.append(fdist_np.most_common(15))\n",
    "    return master_common_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[('trail', 35),\n",
       "  ('plan', 22),\n",
       "  ('use', 19),\n",
       "  ('bicycle', 17),\n",
       "  ('management', 12),\n",
       "  ('cuyahoga', 11),\n",
       "  ('order', 10),\n",
       "  ('executive', 10),\n",
       "  ('information', 9),\n",
       "  ('connector', 7),\n",
       "  ('program', 7),\n",
       "  ('b', 6),\n",
       "  ('park', 6),\n",
       "  ('valley', 6),\n",
       "  ('cvnp', 6)],\n",
       " [('the park', 29),\n",
       "  ('this rule', 16),\n",
       "  ('national park', 9),\n",
       "  ('the superintendent', 9),\n",
       "  (\"the park's\", 7),\n",
       "  ('the cuyahoga', 6),\n",
       "  ('u.s.c. c.', 5),\n",
       "  ('the office', 4),\n",
       "  ('the trail', 4),\n",
       "  ('public notice', 3),\n",
       "  ('old carriage', 3),\n",
       "  ('the nps', 3),\n",
       "  ('public use', 3),\n",
       "  ('executive order', 3),\n",
       "  ('nps gov', 3)],\n",
       " [('the trails forever', 2),\n",
       "  ('the private sector', 2),\n",
       "  ('the old carriage', 2),\n",
       "  ('the national park', 2),\n",
       "  (\"area's natural scenic\", 2),\n",
       "  ('old akron-peninsula road', 1),\n",
       "  ('a critical component', 1),\n",
       "  ('a special regulation', 1),\n",
       "  ('the joint volunteer', 1),\n",
       "  ('the first time', 1),\n",
       "  ('the north side', 1),\n",
       "  ('the presidential memorandum', 1),\n",
       "  ('an open exchange', 1),\n",
       "  ('valley national recreation', 1),\n",
       "  ('the preferred alternative', 1)],\n",
       " [('an important national resource', 1),\n",
       "  ('a significant economic effect', 1),\n",
       "  ('gravel multi-purpose new connector', 1),\n",
       "  (\"the nation's regulatory system\", 1),\n",
       "  ('the east central portion', 1),\n",
       "  ('a major recreational activity', 1),\n",
       "  (\"the park's administrative file\", 1),\n",
       "  ('a 60-day public comment', 1),\n",
       "  ('needed recreational open space', 1)]]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grand_list= get_top_np(np_list)\n",
    "grand_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Lesk algorith for disambiguation in case of multiple synsets of a word\n",
    "def compare_overlaps_greedy(context, synsets_signatures, pos=None):\n",
    "    \"\"\"\n",
    "    Calculate overlaps between the context sentence and the synset_signature\n",
    "    and returns the synset with the highest overlap.\n",
    "    \n",
    "    :param context: ``context_sentence`` The context sentence where the ambiguous word occurs.\n",
    "    :param synsets_signatures: ``dictionary`` A list of words that 'signifies' the ambiguous word.\n",
    "    :param pos: ``pos`` A specified Part-of-Speech (POS).\n",
    "    :return: ``lesk_sense`` The Synset() object with the highest signature overlaps.\n",
    "    \"\"\"\n",
    "    # if this returns none that means that there is no overlap\n",
    "    max_overlaps = 0\n",
    "    lesk_sense = None\n",
    "    for ss in synsets_signatures:\n",
    "        if pos and str(ss.pos()) != pos: # Skips different POS.\n",
    "            continue\n",
    "        overlaps = set(synsets_signatures[ss]).intersection(context)\n",
    "        if len(overlaps) > max_overlaps:\n",
    "            lesk_sense = ss\n",
    "            max_overlaps = len(overlaps)  \n",
    "    return lesk_sense\n",
    "\n",
    "def lesk(context_sentence, ambiguous_word, pos=None, dictionary=None):\n",
    "    \"\"\"\n",
    "    This function is the implementation of the original Lesk algorithm (1986).\n",
    "    It requires a dictionary which contains the definition of the different\n",
    "    sense of each word. See http://goo.gl/8TB15w\n",
    "\n",
    "        >>> from nltk import word_tokenize\n",
    "        >>> sent = word_tokenize(\"I went to the bank to deposit money.\")\n",
    "        >>> word = \"bank\"\n",
    "        >>> pos = \"n\"\n",
    "        >>> lesk(sent, word, pos)\n",
    "        Synset('bank.n.07')\n",
    "    \n",
    "    :param context_sentence: The context sentence where the ambiguous word occurs.\n",
    "    :param ambiguous_word: The ambiguous word that requires WSD.\n",
    "    :param pos: A specified Part-of-Speech (POS).\n",
    "    :param dictionary: A list of words that 'signifies' the ambiguous word.\n",
    "    :return: ``lesk_sense`` The Synset() object with the highest signature overlaps.\n",
    "    \"\"\"\n",
    "    if not dictionary:\n",
    "        dictionary = {}\n",
    "        for ss in wn.synsets(ambiguous_word):\n",
    "            dictionary[ss] = ss.definition().split()\n",
    "    best_sense = compare_overlaps_greedy(context_sentence, dictionary, pos)\n",
    "    return best_sense\n",
    "    #return dictionary \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# this function takes in a word and gets the most relevant synset based on context from the text. \n",
    "# for exact algorith refer the text above (\"what I want to do\" markdown)\n",
    "def get_synset(word, pos):\n",
    "    if len(wn.synsets(word)) == 1:\n",
    "        #print(\"here1\")\n",
    "        return wn.synsets(word)[0]\n",
    "    else:\n",
    "        #get all context sentences\n",
    "        all_sent =[]\n",
    "        for sent in pos_tag_text:\n",
    "            for (w,t) in sent:\n",
    "                if w == word:\n",
    "                    all_sent.append(sent)\n",
    "        #call lesk here\n",
    "        app_syn = lesk(all_sent[len(all_sent)//2], word, pos)\n",
    "        if app_syn != None:\n",
    "            #print(\"here2\")\n",
    "            return app_syn\n",
    "        else:\n",
    "            #second lesk trial with another context sentence\n",
    "            app_syn = lesk(all_sent[len(all_sent)//3], word, pos)\n",
    "            if app_syn != None:\n",
    "                #print(\"here2\")\n",
    "                return app_syn\n",
    "            else:\n",
    "                #give up and choose 1st synset from list with matching pos\n",
    "                #print(\"here3\")\n",
    "                all_syns = wn.synsets(word)\n",
    "                for syn in all_syns:\n",
    "                    #print(syn.pos())\n",
    "                    if syn.pos() == pos:\n",
    "                        return syn\n",
    "    return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# this functions take all the single and double legth phrases form grand_list and gets sysnset for all them. (1 each)\n",
    "def get_singles_doubles_synset():\n",
    "    single_double_synset =[]\n",
    "    #get synsets of all singletons\n",
    "    for (singles,count) in grand_list[0]:\n",
    "        singles_syn = get_synset(singles, 'n')\n",
    "        if singles_syn:\n",
    "            single_double_synset.append(singles_syn)    \n",
    "    #get sysnset of all doubles\n",
    "    for (doubles,count) in grand_list[1]:\n",
    "        doubles_syn = get_synset(doubles.split()[1], 'n')\n",
    "        if doubles_syn:\n",
    "            single_double_synset.append(doubles_syn)\n",
    "    return single_double_synset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Direct concepts\n",
    "def get_direct_concept():\n",
    "    #get all relevant sysnsets\n",
    "    all_synsets = get_singles_doubles_synset()\n",
    "    #extraxting concepts\n",
    "    lemma_names =[]\n",
    "    for synset in all_synsets:\n",
    "        for each_name in synset.lemma_names():\n",
    "            if each_name not in lemma_names:\n",
    "                lemma_names.append(each_name)\n",
    "            break\n",
    "    return lemma_names\n",
    "\n",
    "# mid level concepts\n",
    "def get_general_concepts():\n",
    "    #get all relevant sysnsets\n",
    "    all_synsets = get_singles_doubles_synset()\n",
    "    #get list of their hypernyms\n",
    "    all_hyp = [each_hyp for syn in all_synsets for each_hyp in syn.hypernyms()]\n",
    "    #extraxting general concepts\n",
    "    lemma_names =[]\n",
    "    for synset in all_hyp:\n",
    "        for each_name in synset.lemma_names():\n",
    "            if each_name not in lemma_names:\n",
    "                lemma_names.append(each_name)\n",
    "            break\n",
    "    return lemma_names\n",
    "\n",
    "#get common parents\n",
    "def get_lcs():\n",
    "    #get all relevant sysnsets\n",
    "    all_synsets = get_singles_doubles_synset()\n",
    "    list_of_all_lcs =[]\n",
    "    for syn in all_synsets:\n",
    "        for syn2 in all_synsets[all_synsets.index(syn)+1:]:\n",
    "            lcs = syn.lowest_common_hypernyms(syn2)[0]\n",
    "            if lcs not in list_of_all_lcs:\n",
    "                list_of_all_lcs.append(lcs)\n",
    "    return list_of_all_lcs\n",
    "\n",
    "# get themes\n",
    "def get_theme():\n",
    "    # get common parent\n",
    "    parent_sysnset = get_lcs()\n",
    "    # filter out absolute top level and get lemma_names\n",
    "    lemma_names =[]\n",
    "    for synset in parent_sysnset:\n",
    "        if synset.min_depth() != 0:\n",
    "            #print(synset)\n",
    "            for each_name in synset.lemma_names():\n",
    "                if each_name not in lemma_names:\n",
    "                    lemma_names.append(each_name)\n",
    "                break\n",
    "    return lemma_names\n",
    "    \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "direct_concepts =get_direct_concept()\n",
    "general_concepts = get_general_concepts()\n",
    "themes = get_theme()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Common themes in the text\n",
      "trail\n",
      "plan\n",
      "use\n",
      "bicycle\n",
      "management\n",
      "order\n",
      "executive\n",
      "information\n",
      "connection\n",
      "bacillus\n",
      "park\n",
      "valley\n",
      "rule\n",
      "overseer\n",
      "office\n",
      "notice\n",
      "passenger_car\n",
      "neptunium\n",
      "\n",
      "\n",
      "General concepts in the text\n",
      "path\n",
      "idea\n",
      "activity\n",
      "wheeled_vehicle\n",
      "social_control\n",
      "command\n",
      "administrator\n",
      "message\n",
      "instrumentality\n",
      "eubacteria\n",
      "tract\n",
      "natural_depression\n",
      "concept\n",
      "supervisor\n",
      "place_of_business\n",
      "announcement\n",
      "car\n",
      "metallic_element\n",
      "\n",
      "\n",
      "Direct concepts in the text\n",
      "object\n",
      "location\n",
      "trail\n",
      "physical_entity\n",
      "psychological_feature\n",
      "abstraction\n",
      "plan\n",
      "idea\n",
      "act\n",
      "use\n",
      "whole\n",
      "instrumentality\n",
      "artifact\n",
      "wheeled_vehicle\n",
      "order\n",
      "organism\n",
      "leader\n",
      "message\n",
      "park\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "(len(direct_concepts),len(general_concepts),len(themes))\n",
    "print(\"Common themes in the text\")\n",
    "for word in direct_concepts:\n",
    "    print(word)\n",
    "print('\\n')\n",
    "\n",
    "print(\"General concepts in the text\")\n",
    "for word in general_concepts:\n",
    "    print(word)\n",
    "print('\\n')\n",
    "\n",
    "print(\"Direct concepts in the text\")\n",
    "for word in themes:\n",
    "    print(word)\n",
    "print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Now we will try to create sentence summary from the text. This includes getting sentences from the text that would create a summary for the document **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def create_sentences(corpus):\n",
    "    # First, tokenize the corpus into sentences\n",
    "    sent_tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')\n",
    "    raw_sents = sent_tokenizer.tokenize(corpus)\n",
    "    \n",
    "    return raw_sents\n",
    "\n",
    "def sentence_threshold(sent_list):\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sentences = create_sentences(clean_suppl_info)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}