TetraScience and Databricks Join Forces To Transform Scientific Research, Development, Manufacturing, and Quality Control in Life Sciences BOSTON & SAN FRANCISCO, May 20th, 2024 - TetraScience and Databricks today announced a strategic partnership de...
- 115 Views
- 0 replies
- 2 kudos
yesterday
The ability for organizations to adopt machine learning, AI, and large language models (LLMs) has accelerated in recent years thanks to the popularization of model zoos – public repositories like Hugging Face and TensorFlow Hub that are populated wit...
- 145 Views
- 0 replies
- 1 kudos
yesterday
We are excited to introduce Databricks Assistant Autocomplete now in Public Preview. This feature brings the AI-powered assistant to you in real-time, providing personalized code suggestions as you type. Directly integrated into the notebook and SQL ...
- 103 Views
- 1 replies
- 1 kudos
yesterday
Dive into a world of knowledge with our brand-new Learning Library! Whether you prefer self-paced exploration or guided instruction, our extensive range of courses caters to all personas and learning styles. From beginners to experts, there's someth...
- 2385 Views
- 1 replies
- 0 kudos
a week ago
Community Activity
- 84 Views
- 3 replies
- 1 kudos
Migration Azure to AWS
Hello, today I use Azure Databricks, I want to migrate my wordspaces to AWS Databricks. What is the best practice, which path should I follow?, I didn't find anything in the documentation.thanks.
- 84 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
Hello, as I already have a working Databricks environment on Azure, the best way would be to use tool-databricks-migrate?
- 1 kudos
- 3 Views
- 0 replies
- 0 kudos
To generate DBU consumption report
I need to access the following system tables to generate a DBU consumption report, but I am not seeing this table in the system schema. Could you please help me access it?system.billing.inventory, system.billing.workspaces, system.billing.job_usage, ...
- 3 Views
- 0 replies
- 0 kudos
- 19 Views
- 0 replies
- 0 kudos
A handy tool called spark-column-analyser
I just wanted to share a tool I built called spark-column-analyzer. It's a Python package that helps you dig into your Spark DataFrames with ease.Ever spend ages figuring out what's going on in your columns? Like, how many null values are there, or h...
- 19 Views
- 0 replies
- 0 kudos
- 66 Views
- 2 replies
- 1 kudos
SFTP connection using private key on Azure Databricks
I need to connect to a server to retrieve some files using spark and a private ssh key. However, to manage the private key safely I need to store it as a secret in Azure Key Vault, which means I don't have the key as a file to pass down in the keyFil...
- 66 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @orangepepino, Instead of specifying the keyFilePath, you can pass the private key as a PEM string directly. This approach avoids the need for a physical key file.Since you’re already using Azure Key Vault, consider storing the private key as a s...
- 1 kudos
- 19 Views
- 0 replies
- 0 kudos
Materialized Views GA and Azure Region Availability
Materialized views are currently public preview (as of May 2024). Is there a planned date for GA?Also the limitations section for Azure notes: Databricks SQL materialized views are not supported in the South Central US and West US 2 regions.Will thi...
- 19 Views
- 0 replies
- 0 kudos
- 488 Views
- 1 replies
- 0 kudos
Editing value of widget parameter within notebook code
I have a notebook with a text widget where I want to be able to edit the value of the widget within the notebook and then reference it in SQL code. For example, assuming there is a text widget named Var1 that has input value "Hello", I would want to ...
- 488 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @DavidOBrien, how are you? You can try the following approach: # Get the current value of the widget current_value = dbutils.widgets.get("widget_name") # Append the new value to the current value new_value = current_value + "appended_value" # Se...
- 0 kudos
- 84 Views
- 3 replies
- 0 kudos
Autoloader ignore one folder in path
Hi everyone!I am trying to setup Autoloader to read json file with specific name from all subfolders under the path except one.Could someone advice how this can be achieved? For example, I need to read from .../*/specific_name.json, but ignore test f...
- 84 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
I think you can use REGEXP to achieve this. This might not be the best way, but it should get the job done. It's all about filtering that file in the df from getting loaded. Try something like thisdf.select(“*”,”_metadata”).select(“*”,”_metadata.file...
- 0 kudos
- 1084 Views
- 6 replies
- 0 kudos
log signature and input data for Spark LinearRegression
I am looking for a way to log my `pyspark.ml.regression.LinearRegression` model with input and signature ata. The usual example that I found around are using sklearn and they can simply do # Log the model with signature and input example signature =...
- 1084 Views
- 6 replies
- 0 kudos
- 86 Views
- 7 replies
- 0 kudos
Databricks Autoloader File Notification Not Working As Expected
Hello Everyone,In my project I am using databricks autoloader to incrementally and efficiently processes new data files as they arrive in cloud storage.I am using file notification mode with event grid and queue service setup in azure storage account...
- 86 Views
- 7 replies
- 0 kudos
Latest Reply
- 0 kudos
File notification would only impact any new arriving filesYes, I have all the 3 million files as newly arriving files as I generate synthetic data files for performance testing.From the logs you shared it shows that there are no outstanding bytes or ...
- 0 kudos
- 212 Views
- 1 replies
- 0 kudos
Content Type error legacy serving
Hi,I have deployed an endpoint in Databricks using legacy serving. I am using the custom pyfunc in mlflow to deploy the custom code. This code uses Machine Learning to parse out the table of contents in some pdf files then returns the table of conten...
- 212 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @semsim, Ensure that you’re setting the Content-Type header correctly when making requests to your model endpoint. Since you mentioned using Postman, make sure you set the header to application/json.Verify that the request body is also correctly...
- 0 kudos
- 348 Views
- 3 replies
- 1 kudos
How to find that given Parquet file got imported into Bronze Layer ?
Hi Team,Recently we had created new Databricks project/solution (based on Medallion architecture) having Bronze-Silver-Gold Layer based tables. So we have created Delta-Live-Table based pipeline for Bronze-Layer implementation. Source files are Parqu...
- 348 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
Hello @Devsql , It appears that you are creating DLT bronze tables using a standard spark.read operation. This may explain why the DLT table doesn't include "new files" during a REFRESH operation. For incremental ingestion of bronze layer data into y...
- 1 kudos
- 32 Views
- 0 replies
- 0 kudos
Schema evolution clause added to SQL merge syntax
You can now add the WITH SCHEMA EVOLUTION clause to a SQL merge statement to enable schema evolution for the operation. For more information: https://docs.databricks.com/en/delta/update-schema.html#sql-evo #Databricks
- 32 Views
- 0 replies
- 0 kudos
- 22 Views
- 0 replies
- 0 kudos
VariantType + Parse_json()
In Spark 4.0, there are no more data type mismatches when converting dynamic JSONs, as the new data type VariantType comes with a new function to parse JSONs. Stay tuned for 4.0 release.
- 22 Views
- 0 replies
- 0 kudos
- 109 Views
- 3 replies
- 0 kudos
No Course Materials Widget below Lesson
Hello everyone,In my Databricks partner academy account, there is no course material while it should be under the lesson video. How can I resolve this problem? Does anyone else face the same problem? I had submitted a ticket to ask Databricks team bu...
- 109 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi, @Monsem. I'm sorry about the issue with your Databricks Partner Academy account. Since you've already submitted a ticket without a response, please follow up on your ticket or provide the ticket number. If anyone else has faced this issue and has...
- 0 kudos
- 279 Views
- 4 replies
- 1 kudos
Can't create branch of public git repo
Hi,I have cloned a public git repo into my Databricks account. It's a repo associated with an online training course. I'd like to work through the notebooks, maybe make some changes and updates, etc., but I'd also like to keep a clean copy of it. M...
- 279 Views
- 4 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @DavidKxx , I am not sure I really understand your issue. Can you please provide me the steps you are following to run into this issue so that I can repro it at my end. I referred to the doc, and it appears what you intend is possible. Just refere...
- 1 kudos
Announcements
Latest from our Blog
A guide to quick and scalable historical data loads into Databricks using Azure Data Factory
IntroductionRequirements of a great historical data loadOptionsSolution OverviewTypes of ActivitiesPipeline ParametersPerformanceActivity DetailsCopy activityLoad to tablesValidate tablesOptimize tabl...
Keeping Your Databricks Direct Vector Access Index Fresh in Near Real Time
Keeping your Databricks Direct Vector Access Index fresh in near real time Databricks Vector Search is a vector database that is built into the Databricks Data Intelligence Platform and integrated wit...
How to use System Tables with Overwatch
How to use System Tables with Overwatch Welcome to our blog post on integrating system tables with Overwatch! In this article, we'll delve into the exciting world of leveraging system tables to enhanc...
Retrying dbt Runs in Databricks Workflows
Over the past few years, the variety of tools accessible to data teams has surged, with dbt emerging as a popular solution for data transformation. It empowers SQL-proficient users to craft flexible d...
