cancel
Showing results for 
Search instead for 
Did you mean: 
Page Title

Welcome to the Databricks Community

Discover the latest insights, collaborate with peers, get help from experts and make meaningful connections

104397members
55328posts
cancel
Showing results for 
Search instead for 
Did you mean: 
Get Certified at Data & AI Summit and Earn this Exclusive Databricks Jacket

Agenda at a glance Join us for four days of deep learning around data, AI and LLM technologies. Monday 9:00 AM - 1:00 PM Training and Certification Dive deep into specific topics like data lakehouse architecture, Databricks SQL, MLflow, LLMs and mor...

  • 90 Views
  • 0 replies
  • 1 kudos
18 hours ago
Introducing the Databricks AI Fund

Databricks Ventures Launches New Fund to Extend our Ecosystem Leadership We launched Databricks Ventures in December 2021 as our strategic investment arm for funding innovative startups across the data, analytics and AI landscape — companies that sha...

  • 588 Views
  • 0 replies
  • 0 kudos
Wednesday
Announcing General Availability of Liquid Clustering

Out-of-the-box, self-tuning data layout that scales with your data We’re excited to announce the General Availability of Delta Lake Liquid Clustering in the Databricks Data Intelligence Platform. Liquid Clustering is an innovative data management tec...

  • 669 Views
  • 0 replies
  • 1 kudos
Wednesday
Supercharge Your Code Generation

We are excited to introduce Databricks Assistant Autocomplete now in Public Preview. This feature brings the AI-powered assistant to you in real-time, providing personalized code suggestions as you type. Directly integrated into the notebook and SQL ...

  • 893 Views
  • 1 replies
  • 2 kudos
a week ago

Community Activity

746837
by New Contributor II
  • 1189 Views
  • 3 replies
  • 0 kudos

Databricks and SMTP

Using databricks as aws partner trying to run python script to validate email addresses.  Whenever it gets to the smtp portion it times out.  I am able to telnet from python to the POP servers and get a response, I can ping domains and get replies, b...

  • 1189 Views
  • 3 replies
  • 0 kudos
Latest Reply
Babu_Krishnan
New Contributor III
  • 0 kudos

@746837 , Did you resolve this issue ? 

  • 0 kudos
2 More Replies
prith
by New Contributor III
  • 111 Views
  • 6 replies
  • 1 kudos

Resolved! Datbricks JDK 17 upgrade error

We tried upgrading to JDK 17Using Spark version 3.0.5 and runtime 14.3 LTSGetting this exception using parallelstream()With Java 17 I am not able to parallel process different partitions at the same time.  This means when there is more than 1 partiti...

  • 111 Views
  • 6 replies
  • 1 kudos
Latest Reply
prith
New Contributor III
  • 1 kudos

Anyways - thanks for your response - We found a workaround for this error and JDK 17 is actually working - it appears faster than JDK 8

  • 1 kudos
5 More Replies
Alex42
by New Contributor
  • 171 Views
  • 1 replies
  • 0 kudos

Bug: MLflow connection fails after 2d

Hi there, After exactly 2d of training, the following error is raised after an API call to MLflow:  ValueError: Enum ErrorCode has no value defined for name '403' --------------------------------------------------------------------------- ValueError ...

  • 171 Views
  • 1 replies
  • 0 kudos
Latest Reply
jessysantos
New Contributor III
  • 0 kudos

Hello @Alex42 !The error message indicates that access is forbidden due to an expired access token. This occurs when a notebook or job runs for an extended period, exceeding the default 48-hour threshold set for security reasons. The Databricks acces...

  • 0 kudos
WWoman
by New Contributor III
  • 50 Views
  • 2 replies
  • 0 kudos

Insufficient privileges error when running query from Notebook

Hello all, I am running into a permission issue when running a simple MERGE INTO query from a Notebook: " AnalysisException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have USE SCHEMA on Schema 'system.query'."I can run the que...

  • 50 Views
  • 2 replies
  • 0 kudos
Latest Reply
mhiltner
New Contributor III
  • 0 kudos

Are you using an all purpose cluster or a SQL warehouse to run this query in the notebook?

  • 0 kudos
1 More Replies
Himanshu4
by New Contributor
  • 206 Views
  • 3 replies
  • 2 kudos

Inquiry Regarding Enabling Unity Catalog in Databricks Cluster Configuration via API

Dear Databricks Community,I hope this message finds you well. I am currently working on automating cluster configuration updates in Databricks using the API. As part of this automation, I am looking to ensure that the Unity Catalog is enabled within ...

  • 206 Views
  • 3 replies
  • 2 kudos
Latest Reply
Himanshu4
New Contributor
  • 2 kudos

Hi @raphaelblg ,Thanks for your response. My task is to upgrade jobs. One of the scenarios involves upgrading job clusters, where I am trying to upgrade non-Unity Catalog clusters to Unity Catalog-enabled clusters. Is it possible by upgrading the spa...

  • 2 kudos
2 More Replies
mbhakta
by New Contributor II
  • 1917 Views
  • 3 replies
  • 2 kudos

Change Databricks Connection on Power BI (service)

We're creating a report with Power BI using data from our AWS Databricks workspace. Currently, I can view the report on Power BI (service) after publishing. Is there a way to change the data source connection, e.g. if I want to change the data source...

  • 1917 Views
  • 3 replies
  • 2 kudos
Latest Reply
Srushti
Visitor
  • 2 kudos

Have you got any solution for this?

  • 2 kudos
2 More Replies
Kutbuddin
by New Contributor
  • 177 Views
  • 5 replies
  • 0 kudos

Resolved! Stream Query termination using available now trigger and toTable.

We are running a streaming job in databricks with custom streaming logic which consumes a CDC stream from mongo and appends to a delta table, at the end of the streaming job we have a internal checkpointing logic which creates an entry into a table w...

  • 177 Views
  • 5 replies
  • 0 kudos
Latest Reply
Kutbuddin
New Contributor
  • 0 kudos

I was expecting spark.sql(f"insert into table {internal_tab_name} values({dt})") to execute at the end after the streaming query was written to the table. What I observed:The spark sql query spark.sql(f"insert into table {internal_tab_name} values({d...

  • 0 kudos
4 More Replies
DouglasMoore
by New Contributor II
  • 43 Views
  • 0 replies
  • 0 kudos

How to enable unity catalog system tables?

Unity Catalog system tables provide lots of metadata & log data related to the operations of Databricks. System tables are organized into separate schemas containing one to a few tables owned and updated by Databricks. The storage and the cost of the...

  • 43 Views
  • 0 replies
  • 0 kudos
surband
by New Contributor III
  • 143 Views
  • 1 replies
  • 0 kudos

Databricks Run Notebook GitHub Action

The GitHub action databricks/run-notebook to deploy and run a notebook from GitHub to DBX awaits the completion of the Job. The pulsar streaming job that I have is a long running job due to which the Action times out when the access token it uses to ...

  • 143 Views
  • 1 replies
  • 0 kudos
Latest Reply
surband
New Contributor III
  • 0 kudos

https://github.com/databricks/run-notebook/issues/53#issue-2321682696

  • 0 kudos
totaltools
by Visitor
  • 35 Views
  • 0 replies
  • 0 kudos

Optimizing Data Pipelines: The Right Tools for the Spark in Your Warehouse

Hey there, fellow data enthusiasts! In the world of data warehousing and analytics, building efficient data pipelines is crucial. While the power of Apache Spark is undeniable, having the right tools at your disposal can significantly streamline the ...

  • 35 Views
  • 0 replies
  • 0 kudos
190809
by Contributor
  • 989 Views
  • 3 replies
  • 2 kudos

Is there a way to add a date parameter to the jobs run API call?

Hi there I am currently making a call to the Databricks API jobs run endpoint. I would like to make this call on a daily basis to get data on the jobs run in the past 24 hours and add this to my delta table. Is there a way to set a GTE value in the A...

  • 989 Views
  • 3 replies
  • 2 kudos
Latest Reply
AdrianC
Visitor
  • 2 kudos

Actually the "start_time_to" parameter doesn't seem to work at all. Neither alone or together with "start_time_from" (Whenever used the api call returns nothing). I'd like to report this as an issue as we want to automate our cluster usage monitoring...

  • 2 kudos
2 More Replies
aranjan99
by New Contributor
  • 176 Views
  • 3 replies
  • 0 kudos

system.billing.usage table missing data for jobs running in my databricks account

I have some jobs running on databricks. I can obtain their jobId from the Jobs UI or List Job Runs API.However when trying to get DBU usage for the corresponding jobs from system.billing.usage, I do not see the same job_id in that table. Its been mor...

  • 176 Views
  • 3 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @aranjan99, I understand your concern. The system.billing.usage table in Databricks is updated every hour, and it only includes completed tasks1. If your jobs are long-running and have not completed yet, they will not appear in this table1. Add...

  • 0 kudos
2 More Replies
liliana_tang
by New Contributor III
  • 388 Views
  • 0 replies
  • 0 kudos

Edmonton - Data & AI Summit Watchparty

Hello everyone! Can't make it to San Francisco this year to attend the data and AI Summit? No problem, we have a watchparty in Edmonton, so you can experience the latest trends in data and artificial intelligence with like-minded individuals locally....

  • 388 Views
  • 0 replies
  • 0 kudos
laurappp
by Visitor
  • 58 Views
  • 2 replies
  • 0 kudos

Deploy ML

How deploy a model using community account version?

  • 58 Views
  • 2 replies
  • 0 kudos
Latest Reply
Walter_C
Valued Contributor II
  • 0 kudos

From your workspace page are you able to see the option to create Models?  

  • 0 kudos
1 More Replies
phguk
by New Contributor III
  • 263 Views
  • 2 replies
  • 0 kudos

Efficient methods to make a temporary copy of a table

I'm using a tool (SAS) that doesn't inherently support time travel - that's to say it doesn't generate SQL including Timestamp or Version (for example). An obvious work-around could be to first copy/clone the version of the table, which SAS can then ...

  • 263 Views
  • 2 replies
  • 0 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

@phguk I think that Shallow Clone would be the best solution here.

  • 0 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Top Kudoed Authors

Latest from our Blog

How to use Databricks Autoloader across AWS accounts

IntroductionCreate S3 bucket and cross-account instance profileUse Autoloader to create SNS-SQS across accountsManually create SNS-SQS for cross-account AutoloaderTest cross-account Autoloader connect...

57Views 1kudos

Grouped Pandas Optimization

In this short tutorial, we’ll implement an approach to making certain applyInPandas operations run many times faster. First, let's generate some dummy data for this example using Spark. For our exampl...

69Views 1kudos