Skip to content
  • Texas | Arizona | Virginia | Idaho | Illinois
  • (888) 705-0930
  • info@therawragency.com
Facebook-f Twitter Instagram Linkedin-in
rawr logo short
  • Home
  • About
  • Solutions
    colorized icons 04
    B2B Marketing
    colorized icons 05
    B2C Marketing
    colorized icons 06
    Online Visibility Management
  • Industries
    colorized icons 07
    Healthcare
    colorized icons 08
    Manufacturing
    colorized icons 10
    Home Services
    colorized icons 09
    Professional Services (B2B)
    colorized icons 11
    Retail
  • Services
    colorized icons 01
    Brand Strategy
    • Graphic Design
    • Corporate Brand Identity
    • Sales Enablement
    • Editorial Calendar Management
    colorized icons 02
    Website Design
    • WordPress Website Development
    • Conversion Rate Optimization
    • eCommerce Development
    • Content Strategy & Copywriting
    colorized icons 03
    Digital Marketing
    • Digital Marketing Strategy
    • Search Engine Optimization (SEO)
    • Pay-Per-Click (PPC) Management
    • Content Marketing & Digital PR
    • Account Based Marketing (B2B)
    • Marketing Automation
    • Social Media Management
  • Blog
  • Contact
Let's Talk

Google opens the source for its robots.txt parser in Java and testing framework in C++

  • September 23, 2020
  • General
  • Agency
Facebook
Twitter
LinkedIn
Email

Last year, Google open sourced the code for the robots.txt parser used in its production systems. After seeing the community build tools with it and add their own contributions to the open source library, including language ports of the original parser written in C++ to golang and rust, Google announced this week it has released additional related source code projects.

Here’s what’s new for developers and tech SEOs to play with.

C++ and Java. For anyone writing their own or adopting Google’s parser written in C++ (a super fast compiled language), Google has released the source code for its robots.txt parser validation testing framework used to ensure parser results adhere to the official robots.txt specification as expected, and it can validate parsers written in a wide variety of other languages.

Additionally, Google released an official port to the more popular Java language. Modern Java is more widely used in enterprise applications than C++, whereas C++ is more typically used in core system applications where performance needs demand it. Some Java-based codebases run applications today for enterprise SEO and or marketing software.

Testing and validation. Requirements for running the test framework include JDK 1.7+ for Apache Maven, and Google’s protocol buffer to interface the test framework with your parser platform and development workstation. It should be useful to anyone developing their own parser, validating a port, or utilizing either of Google’s official parsers, and especially for validating your development of a port to a new language.

How difficult would this be to use? We should note these are relatively approachable intern-led projects at Google which ought to be consumable by moderate to higher level programmers in one or more of these languages. You can build a robots.txt parser using practically any programming language. It adds perceived authority, however, when your marketing application runs the exact same parser that governs Googlebot.

Why we care. If you, or your company, has plans to write or has written a crawler which parses robots.txt files for directives looking for important information (not just) for SEO, then this gives you incentive to evaluate whether using Google’s parser in C++, Java, or one of the other language ports is worth it. The Java parser in particular should be relatively easy to adopt if your application is already written in Java.

The post Google opens the source for its robots.txt parser in Java and testing framework in C++ appeared first on Search Engine Land.


Source: IAB

Facebook
Twitter
LinkedIn
Email

Recent Blog Posts

9 Hidden Revenue Blockers Every CRO Needs to Eliminate

February 3, 2025

What Is B2B Appointment Setting?

November 29, 2024

Leveraging LinkedIn for Targeted Growth

November 12, 2024

Mastering B2B Prospecting: Key Strategies for Sales Success

November 4, 2024

Maximize Marketing ROI: How Cost per Lead Can Fuel Your Business Growth

June 13, 2024
View More
rawr logo short
Facebook-f Twitter Instagram Linkedin-in
Get In Touch
  • Texas | Arizona | Virginia | Idaho | Illinois
  • (888) 705-0930
  • info@therawragency.com
Send An Email

"*" indicates required fields

Copyright 2025 | The RAWR Agency, LLC. |

Sitemap | Privacy Policy

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.OkNo