Run Scala, Python and SQL workloads on shared, cost-efficient multi-user compute. We are thrilled to announce Unity Catalog Lakeguard, which allows you to run Apache Spark™ workloads in SQL, Python, and Scala with full data governance on the Databric...
- 130 Views
- 1 replies
- 1 kudos
yesterday
Get started with Databricks Asset Bundles (DABs) today! Simplify project management, versioning, testing, and deployment for your data and AI projects on the Databricks Platform. Embrace software engineering best practices with DABs—source control, c...
- 665 Views
- 0 replies
- 2 kudos
Wednesday
For a limited time, we're offering 50% off training and certification at Data + AI Summit with the following code: TRAIN50FOTY. This offer expires on May 3, 2024. Register today and add training. Read more here!
- 221 Views
- 0 replies
- 0 kudos
Tuesday
The Big Book of Generative AI brings together best practices and know-how for building production-quality GenAI applications. You’ll find technical content and code samples that will help you do everything from deploying your first application to bui...
- 616 Views
- 1 replies
- 2 kudos
Monday
Community Activity
- 1 Views
- 0 replies
- 0 kudos
Testosil Reviews (2024): Is It Worth Buying Or Not
Testosil rocks! It's been a total game-changer for me. Since I started using it, I've felt a real boost in my sprightliness and focus. What's awesome is that it's made with natural ingredients, so I feel good about what I'm putting into my body. Plus...
- 1 Views
- 0 replies
- 0 kudos
- 0 Views
- 0 replies
- 0 kudos
Impersonating a user
How do I impersonate a user? I can't find any documentation that explains how to do this or even hint that it's possible.Use case: I perform administrative tasks like assign grants and roles to catalogs, schemas, and tables for the benefit of busines...
- 0 Views
- 0 replies
- 0 kudos
- 4 Views
- 0 replies
- 0 kudos
Testosil Reviews (April 2024): Is It Worth or Not
Testosil delivers a powerhouse punch for vitality seekers. Within days of starting, I noticed a remarkable surge in energy levels, allowing me to power through the day with vigor. Not only did it improve my physical strength, but it also uplifted my ...
- 4 Views
- 0 replies
- 0 kudos
- 20 Views
- 0 replies
- 0 kudos
Notebook cell gets hung up but code completes
Have been running into an issue when running a pymc-marketing model in a Databricks notebook. The cell that fits the model gets hung up and the progress bar stops moving, however the code completes and dumps all needed output into a folder. After the...
- 20 Views
- 0 replies
- 0 kudos
- 11520 Views
- 8 replies
- 3 kudos
Resolved! Private PyPI repos on DBR 13+
We use a private PyPI repo (AWS CodeArtifact) to publish custom python libraries. We make the private repo available to DBR 12.2 clusters using an init-script as prescribed here in the Databricks KB. When we tried to upgrade to 13.2 this stopped wor...
- 11520 Views
- 8 replies
- 3 kudos
Latest Reply
- 3 kudos
I'm coming back to provide an updated solution that doesn't rely on the implementation detail of the user name (e.g., libraries) - which is not considered a contract and could potentially change and break in the future.The key is to use the --global ...
- 3 kudos
- 5386 Views
- 8 replies
- 8 kudos
Resolved! Installing libraries on job clusters
Simple question : what is the way to go to install libraries on job clusters ? There does not seem to be a "Libraries" tab on the UI as opposed to regular clusters. Does it mean that the only option is to use init scripts ?
- 5386 Views
- 8 replies
- 8 kudos
Latest Reply
- 8 kudos
@daniel_sahalflappy birdIs there another way to add libraries to the ADF Databricks task even faster?
- 8 kudos
- 29 Views
- 0 replies
- 0 kudos
Insufficient Permissions Issue on Databricks
I have encountered a technical issue on Databricks.While executing commands both in Spark and SQL within the Databricks environment, I’ve run into permission-related errors from selecting files from DBFS. "org.apache.spark.SparkSecurityException: [IN...
- 29 Views
- 0 replies
- 0 kudos
- 19 Views
- 0 replies
- 0 kudos
Unable to access unity catalog volume via /Volumes in notebook
I have set up a volume in unity catalog in the format catalog/schema/volume, and granted all permissions to all users on the catalog, schema and volume.From the notebook I can see the /Volumes directory in the root of the file system but am unable to...
- 19 Views
- 0 replies
- 0 kudos
- 10 Views
- 0 replies
- 0 kudos
udf in databricks
Hi Team,Is there a particular reason why we should avoid using UDF and instead convert to DataFrame code?Are there any restrictions or limitations (in terms of performance or governance) when using UDFs in Databricks? Regards,Janga
- 10 Views
- 0 replies
- 0 kudos
- 25 Views
- 0 replies
- 0 kudos
IP address for accessing external SFTP server
I am trying to pull in data to my Databricks workspace via an external SFTP server. I am using Azure for my compute. To access the SFTP server they need to whitelist my IP address. My IP address in Azure Databricks seems to be constantly changing fro...
- 25 Views
- 0 replies
- 0 kudos
- 701 Views
- 2 replies
- 1 kudos
Resolved! DELTA_EXCEED_CHAR_VARCHAR_LIMIT
Hi,I am having an issue of loading source data into a delta table/ unity catalog. The error we are recieving is the following:grpc_message:"[DELTA_EXCEED_CHAR_VARCHAR_LIMIT] Exceeds char/varchar type length limitation. Failed check: (isnull(\'metric_...
- 701 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hey @Paul92S Looking at the error message it looks like column "metric_name" is the culprit here:Understanding the Error:Character Limit Violation: The error indicates that values in the metric_name column are exceeding the maximum length allowed fo...
- 1 kudos
- 375 Views
- 2 replies
- 1 kudos
Query results in csv file include 'null' string for blank cell
After running a sql script, when downloading the results to a csv file, the file includes a null string for blank cells (see screenshot). Is ther a setting I can change to simply get empty cells instead?
- 375 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Thank you. How would you download to csv the results from a SQL script without the null strings in empty cells?I mainly work with SQL notebooks
- 1 kudos
- 21 Views
- 0 replies
- 0 kudos
Pass Dataframe to child job in "Run Job" task
Hello,I have a Job A that runs a Job B, and Job A defines a globalTempView and I would like to somehow access it in the child job. Is that in anyway possible ? Can the same cluster be used for both jobs ? If it is not possible, does someone know of a...
- 21 Views
- 0 replies
- 0 kudos
- 21 Views
- 0 replies
- 0 kudos
www.mca.gov.in- Ministry Of Corporate Affairs
Today, the entire private limited company registration process and other regulatory filings are paperless; documents are filed electronically through the MCA website and is processed at the Central Registration Centre (CRC). The Online Private Limite...
- 21 Views
- 0 replies
- 0 kudos
- 4360 Views
- 10 replies
- 6 kudos
Databricks now supports event-driven workloads, especially for loading cloud files from external locations. This means you can save costs and resource...
Databricks now supports event-driven workloads, especially for loading cloud files from external locations. This means you can save costs and resources by triggering your Databricks jobs only when new files arrive in your cloud storage instead of mou...
- 4360 Views
- 10 replies
- 6 kudos
Latest Reply
- 6 kudos
@daniel_sahal I get your point, but if for a scheduled trigger you can get all kind of attributes on the trigger time (arguably, this is available for all the triggers), then why wouldn't the most important attribute of a file event not be available ...
- 6 kudos
Announcements
Latest from our Blog
How to enhance data protection in Azure Databricks with Confidential Computing
Proposed by Databricks in 2020, the Lakehouse architecture has increasingly been embraced by the industry in the past years. Databricks has since applied Generative AI and evolved the Databricks Lakeh...
Prepare Your Journey to Migrate from AWS Glue Data Catalog to Databricks Unity Catalog
In this blog, we will look at the migration from AWS Glue Data Catalog to Unity Catalog. We cover how to plan this migration as a step-by-step approach and emphasize meticulous planning, phased migrat...
Mastering the Spark UI
Almost anyone who uses Spark or Databricks is aware of the Spark UI, and we all know that it’s a super powerful tool in the right hands. It can reveal what’s going wrong and any inefficiencies in you...
Optimize costs for your Data and AI workloads with Azure and AWS Spot Instances
Introduction Cost optimisation remains a pivotal challenge for customers dealing with processing large volumes of data and machine learning model training at scale in the cloud. Spot instances have re...
