Get Started Resources
0 Posts
Activity in Resources
- 52 Views
- 1 replies
- 0 kudos
A handy tool called spark-column-analyser
I just wanted to share a tool I built called spark-column-analyzer. It's a Python package that helps you dig into your Spark DataFrames with ease.Ever spend ages figuring out what's going on in your columns? Like, how many null values are there, or h...
- 52 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
An example added to README in GitHubDoing analysis for column PostcodeJson formatted output{"Postcode": {"exists": true,"num_rows": 93348,"data_type": "string","null_count": 21921,"null_percentage": 23.48,"distinct_count": 38726,"distinct_percentage"...
- 0 kudos
- 491 Views
- 1 replies
- 0 kudos
Editing value of widget parameter within notebook code
I have a notebook with a text widget where I want to be able to edit the value of the widget within the notebook and then reference it in SQL code. For example, assuming there is a text widget named Var1 that has input value "Hello", I would want to ...
- 491 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @DavidOBrien, how are you? You can try the following approach: # Get the current value of the widget current_value = dbutils.widgets.get("widget_name") # Append the new value to the current value new_value = current_value + "appended_value" # Se...
- 0 kudos
- 42 Views
- 0 replies
- 0 kudos
Schema evolution clause added to SQL merge syntax
You can now add the WITH SCHEMA EVOLUTION clause to a SQL merge statement to enable schema evolution for the operation. For more information: https://docs.databricks.com/en/delta/update-schema.html#sql-evo #Databricks
- 42 Views
- 0 replies
- 0 kudos
- 40 Views
- 0 replies
- 0 kudos
VariantType + Parse_json()
In Spark 4.0, there are no more data type mismatches when converting dynamic JSONs, as the new data type VariantType comes with a new function to parse JSONs. Stay tuned for 4.0 release.
- 40 Views
- 0 replies
- 0 kudos
- 115 Views
- 1 replies
- 1 kudos
Supercharge Your Code Generation
We are excited to introduce Databricks Assistant Autocomplete now in Public Preview. This feature brings the AI-powered assistant to you in real-time, providing personalized code suggestions as you type. Directly integrated into the notebook and SQL ...
- 115 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Great features, help to write code much faster
- 1 kudos
- 126 Views
- 0 replies
- 2 kudos
Accelerating the Scientific AI Revolution
TetraScience and Databricks Join Forces To Transform Scientific Research, Development, Manufacturing, and Quality Control in Life Sciences BOSTON & SAN FRANCISCO, May 20th, 2024 - TetraScience and Databricks today announced a strategic partnership de...
- 126 Views
- 0 replies
- 2 kudos
- 171 Views
- 0 replies
- 1 kudos
Deploying Third-party models securely with the Databricks Data Intelligence Platform and HiddenLayer
The ability for organizations to adopt machine learning, AI, and large language models (LLMs) has accelerated in recent years thanks to the popularization of model zoos – public repositories like Hugging Face and TensorFlow Hub that are populated wit...
- 171 Views
- 0 replies
- 1 kudos
- 156 Views
- 0 replies
- 0 kudos
Getting Started with Databricks - Building a Forecasting Model on Databricks
Building a Forecasting Model on Databricks: A Step-by-Step Guide This guide offers a detailed, step-by-step approach for building a forecasting model on Databricks. By leveraging the power of Databricks, you will unlock new potentials in your data w...
- 156 Views
- 0 replies
- 0 kudos
- 28133 Views
- 52 replies
- 111 kudos
🔔 Attention Databricks Academy Users: SSO Implementation Incoming! Secure Your Account Today!
On May 18th a new SSO user experience will launch in Customer and Partner Academy. In order to secure your account, please see the instructions below on how to update and verify that your data is correct and complete. Please ensure that you have two ...
- 28133 Views
- 52 replies
- 111 kudos
Latest Reply
- 111 kudos
I'm a consultant. I work with multiple companies, but I don't have a main one. I'm not comfortable setting any of my emails in clients' domains on my profile. Is it OK for me to keep using my personal one?
- 111 kudos
- 516 Views
- 0 replies
- 0 kudos
Building DBRX-class Custom LLMs with Mosaic AI Training
We recently introduced DBRX: an open, state-of-the-art, general-purpose LLM. DBRX was trained, fine-tuned, and evaluated using Mosaic AI Training, scaling training to 3072 NVIDIA H100s and processing more than 12 trillion tokens in the process. Train...
- 516 Views
- 0 replies
- 0 kudos
- 856 Views
- 0 replies
- 1 kudos
Accurate, Safe and Governed: How to Move GenAI from POC to Production
In the realm of AI, achieving accuracy is paramount. The publication delves into techniques for refining models to ensure they reliably deliver precise outcomes in real-world scenarios. It covers methodologies such as continuous monitoring, data augm...
- 856 Views
- 0 replies
- 1 kudos
- 610 Views
- 0 replies
- 0 kudos
Getting Started with Databricks - Ingest to Analytics & BI
Getting Started with Databricks - From Ingest to Analytics & BIIntroduction to DatabricksAnalytics & BI on DatabricksIngest Setup Steps [20 minutes]Step 0: Check your required prerequisitesStep 1: Access and start your warehouseStep 2: Connect your w...
- 610 Views
- 0 replies
- 0 kudos
- 189 Views
- 1 replies
- 0 kudos
How to convert txt files to delta tables
Hello members of Databricks's comunity,I am currently working on a project where we collect data from machines, that data is in .txt format. The data is currently in an Azure container, I need to clean the files and convert them to delta tables, how ...
- 189 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
https://docs.databricks.com/en/ingestion/add-data/upload-data.html
- 0 kudos
- 120 Views
- 0 replies
- 0 kudos
RocksDB for storing state stream
Now, you can keep the state of stateful streaming in RocksDB. For example, retrieving keys from memory to check for duplicate records inside the watermark is now faster. #databricks
- 120 Views
- 0 replies
- 0 kudos
- 291 Views
- 1 replies
- 0 kudos
Understand why your jobs' performances are changing over time
Hi Folks -We released a new metrics view for databricks jobs in Gradient, which helps track and plot the metrics below over time to help engineers understand what's going on with their jobs over time.Job cost (DBU + Cloud fees)Job RuntimeNumber of co...
- 291 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @data_turtle, That sounds like a valuable addition to Gradient! The new metrics view for Databricks jobs will surely help engineers gain better insights into their job performance and resource usage over time. Being able to track metrics such as j...
- 0 kudos
