Run CancerGov Scrape and Store — cg

Run the full sequence that scrapes, parses, and stores the NCI Drug Dictionary found at CancerGov.org and any correlates to the NCI Thesaurus in a Postgres Database.

cg_run(
  conn,
  conn_fun,
  steps = c("log_drug_count", "get_dictionary_and_links", "process_drug_link_synonym",
    "process_drug_link_url", "process_drug_link_ncit", "get_ncit_synonym",
    "update_cancergov_drugs"),
  max_page = 50,
  sleep_time = 5,
  encoding = "",
  options = c("RECOVER", "NOERROR", "NOBLANKS"),
  expiration_days = 30,
  verbose = TRUE,
  render_sql = TRUE
)

Arguments

conn	Postgres connection object.
conn_fun	(optional) An expr as a string that can be parsed and evaluated into a connection object. If present, it is used in lieu of the `conn` argument and disconnects the connection on exit.
steps	The sequence of steps, labeled by the internal function that is called. A step can be skipped if it is removed from the list, but the order in which they are called does not change. Adding any erroneous values to this list does not have an effect. Default: c("log_drug_count", "get_dictionary_and_links", "process_drug_link_synonym", "process_drug_link_url", "process_drug_link_ncit", "get_ncit_synonym", "update_cancergov_drugs").
max_page	maximum page number to iterate the scrape over in the "https://www.cancer.gov/publications/dictionaries/cancer-drug?expand=ALL&page=" path, Default: 50
sleep_time	Time in seconds for the system to sleep before each scrape with `read_html`.
encoding	Specify a default encoding for the document. Unless otherwise specified XML documents are assumed to be in UTF-8 or UTF-16. If the document is not UTF-8/16, and lacks an explicit encoding directive, this allows you to supply a default.
options	Set parsing options for the libxml2 parser. Zero or more of RECOVER recover on errors NOENT substitute entities DTDLOAD load the external subset DTDATTR default DTD attributes DTDVALID validate with the DTD NOERROR suppress error reports NOWARNING suppress warning reports PEDANTIC pedantic error reporting NOBLANKS remove blank nodes SAX1 use the SAX1 interface internally XINCLUDE Implement XInclude substitition NONET Forbid network access NODICT Do not reuse the context dictionary NSCLEAN remove redundant namespaces declarations NOCDATA merge CDATA as text nodes NOXINCNODE do not generate XINCLUDE START/END nodes COMPACT compact small text nodes; no modification of the tree allowed afterwards (will possibly crash if you try to modify the tree) OLD10 parse using XML-1.0 before update 5 NOBASEFIX do not fixup XINCLUDE xml:base uris HUGE relax any hardcoded limit from the parser OLDSAX parse using SAX2 interface before 2.7.0 IGNORE_ENC ignore internal document encoding hint BIG_LINES Store big lines numbers in text PSVI field
verbose	When reading from a slow connection, this prints some output on every iteration so you know its working.

Web Source Types

The NCI Drug Dictionary has 2 data sources that run in parallel. The first source is the Drug Dictionary itself at https://www.cancer.gov/publications/dictionaries/cancer-drug. The other source are the individual drug pages, called Drug Detail Links in skyscraper, that contain tables of synonyms, including investigational names.

Drug Dictionary

The listed drug names and their definitions are scraped from the Drug Dictionary HTML and updated to a Drug Dictionary Table in a cancergov schema.

Drug Detail Links

The links to Drug Pages are scraped from the Data Dictionary URL over the maximum page number and are saved to a Drug Link Table in the cancergov schema. The URLs in the Drug Link Table are then scraped for any HTML Tables of synonyms and the results are written to a Drug Link Synonym Table. The links to active clinical trials and NCIt mappings are also derived and stored in their respective tables.