cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Karl
by New Contributor II
  • 19636 Views
  • 2 replies
  • 3 kudos

PySpark column object not callable using "when otherwise" transformation

The very first "when" function results in the posted error message (see image). The print statement of the count of df_td_amm works. A printSchema of the "df_td_amm" data frame confirms that "AGE" is a column. A select statement is also successful, s...

Error
  • 19636 Views
  • 2 replies
  • 3 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 3 kudos

the syntax is when(....).otherwise(...), not other(...)And there are some backslashes missing.

  • 3 kudos
1 More Replies
vk217
by Contributor
  • 1157 Views
  • 0 replies
  • 0 kudos

Databricks migration from 7.3 LTS to X.x

We are currently on 7.3 LTS version with Python 3.7. I see that we are several versions off the latest 11.1 release https://docs.databricks.com/release-notes/runtime/releases.html.I see that the End of support for 11.1 is earlier than 10.4 LTS. What ...

  • 1157 Views
  • 0 replies
  • 0 kudos
antoniodavideca
by New Contributor III
  • 4891 Views
  • 5 replies
  • 1 kudos

Resolved! Jobs REST Api - Run a Job that is connected to a git_source

On Jobs REST API is possible to create a new Job, specifying a git_source.My question is about triggering the job.Still on Jobs REST Api is possible to trigger a job using the job_id, but I don't find a way to tell anyhow to Databricks, what's the en...

  • 4891 Views
  • 5 replies
  • 1 kudos
Latest Reply
Prabakar
Databricks Employee
  • 1 kudos

Ah. Got it. So is your issue resolved or are you looking for further information.

  • 1 kudos
4 More Replies
Gabriel0007
by New Contributor III
  • 2814 Views
  • 0 replies
  • 3 kudos

How to save json data to Delta Table: ParseError on Insert

I'm trying to save the returned json data from a requests API call to a Delta Table. I get a ParseError when I INSERT the response object which is in json format. The error shows the json data and a marker that states a ' or } or ) is missing. I v...

  • 2814 Views
  • 0 replies
  • 3 kudos
Kit
by New Contributor III
  • 6518 Views
  • 7 replies
  • 1 kudos

Resolved! Can't run a job that use GitHub as source

I have a list of jobs that are using the code in GitHub as source.Everything worked fine until yesterday. Yesterday, I noticed that all the job that were using GitHub as source were failing. Because of the following error: ``` Run result unavailable:...

  • 6518 Views
  • 7 replies
  • 1 kudos
Latest Reply
User16766737456
Databricks Employee
  • 1 kudos

Just an update, to round this out. We investigated further internally, and found that although we have a cleanup process in place to remove the internal repos that are being checked out for workflows, it was failing to catch up due to the sheer volum...

  • 1 kudos
6 More Replies
antoniodavideca
by New Contributor III
  • 3380 Views
  • 2 replies
  • 0 kudos

Jobs REST Api - Create new Job with a new Cluster, and install a Maven Library on the Cluster

I would need to use the Job REST API to create a Job on our databrick Cluster.At the Job Creation, is possible to specify an existing cluster, or, create a new one.I can forward alot of information to the Cluster, but what I would like to specify is ...

  • 3380 Views
  • 2 replies
  • 0 kudos
Latest Reply
Prabakar
Databricks Employee
  • 0 kudos

@Antonio Davide Cali​ You can use the existing cluster in your json to use it for the job.To update or push libraries to the job, you can use the JobsUpdate API. As you want to push libraries to the cluster, you can push them using the new setting an...

  • 0 kudos
1 More Replies
Lazloo
by New Contributor III
  • 1650 Views
  • 0 replies
  • 2 kudos

Cannot load spark-avro jars with databricksversion 10.4

Currently, I am facing an issue since the `databricks-connect` runtime on our cluster was updated to 10.4. Since then, I cannot load the jars for spark-avro anymore. By Running the following code from pyspark.sql import SparkSession   spark = SparkSe...

  • 1650 Views
  • 0 replies
  • 2 kudos
Anonymous
by Not applicable
  • 875 Views
  • 0 replies
  • 5 kudos

www.databricks.com

New and Exciting! Databricks and Jupyter: Announcing ipywidgets in the Databricks NotebookBringing the interactivity of the Jupyter ecosystem into the LakehouseWe are excited to announce a deeper integration between the Databricks Notebook and the e...

  • 875 Views
  • 0 replies
  • 5 kudos
sauri
by New Contributor III
  • 4131 Views
  • 6 replies
  • 4 kudos

Hello, I runn a notebook with datafactory and I have this error:Failure starting repl. Try detaching and re-attaching the notebook. But my notebook is...

Hello, I runn a notebook with datafactory and I have this error:Failure starting repl. Try detaching and re-attaching the notebook.But my notebook is attachingCan I attach notebook with a comand o r code ?

  • 4131 Views
  • 6 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hey @oscar alvarez​ Thank you so much for getting back to us. It's really great of you to send in the solution. Would you be happy to mark it as best so that other members can find the solution more quickly?We really appreciate your time.Wish you a g...

  • 4 kudos
5 More Replies
Kara
by New Contributor II
  • 1180 Views
  • 0 replies
  • 1 kudos

Integrating Databricks repos with Azure DevOps

Hi, Databricks community. I am trying to integrate Databricks shared folder notebooks with Azure DevOps GIT repositories. Can someone please point me to a basic training tutorial (or video) on how to get started and best practices?

  • 1180 Views
  • 0 replies
  • 1 kudos
yopbibo
by Contributor II
  • 3607 Views
  • 2 replies
  • 0 kudos

Resolved! Cluster configuration / notebook panel

Hi,Is it possible to let regular users to see all running notebooks (in the notebook panel of the cluster) on a specific cluster they can use (attach and restart).by default admins can see all running notebooks and users can see only their own notebo...

  • 3607 Views
  • 2 replies
  • 0 kudos
Latest Reply
Prabakar
Databricks Employee
  • 0 kudos

hi @Philippe CRAVE​ a user can see a notebook only if they have permission to that notebook. Else they won't be able to see it. Unfortunately there is no possibility for a normal user to see the notebooks attached to a cluster if they do not have per...

  • 0 kudos
1 More Replies
akshay1
by New Contributor II
  • 2613 Views
  • 0 replies
  • 2 kudos

Data unloading to S3 bucket from Databricks.

Hi,I am completely new to the Databricks & have a task to unload the data from Databricks table to the S3 location using java/sql. Is this possible? If yes can you please help me?

  • 2613 Views
  • 0 replies
  • 2 kudos
User16790091296
by Contributor II
  • 6184 Views
  • 1 replies
  • 2 kudos

How to restart a cluster on databricks using databricks-CLI?

I'm trying to restart an existing cluster in Databricks on Azure using databricks-cli.I'm using the following command:databricks clusters restart {"cluster_id": "0710-121255-liner30"}But it gives giving me this error:Error: Missing option "--cluster-...

  • 6184 Views
  • 1 replies
  • 2 kudos
Latest Reply
User16766737456
Databricks Employee
  • 2 kudos

Can you try:databricks clusters restart --cluster-id <the-cluster-id>$ databricks clusters restart --help Usage: databricks clusters restart [OPTIONS]   Restarts a Databricks cluster given its ID.   If the cluster is not currently in a RUNNING st...

  • 2 kudos
huggies_23
by New Contributor
  • 1488 Views
  • 0 replies
  • 0 kudos

Is it possible to specify a specific branch commit when deploying repo to a workspace via the Databricks CLI?

I would like to know if it is possible to include a specific commit identifier when updating a repo in a workspace via the Databricks CLI.Why? Currently we use the repos CLI to push updates to code throughout dev, test and prod (testing along the wa...

  • 1488 Views
  • 0 replies
  • 0 kudos
Taha_Hussain
by Databricks Employee
  • 20032 Views
  • 2 replies
  • 6 kudos

Resolved! Create a Dashboard: How do I visualize data with Databricks SQL or my BI tool?

Databricks SQL helps query and visualize data so you can share real-time business insights with built-in dashboards or your favorite BI tools.This post helps you create queries, visualizations and dashboards and connect to your BI tools for deeper da...

Databricks SQL Locked DBSQL Create A Query Data Explorer
  • 20032 Views
  • 2 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Thanks for the information, I will try to figure it out for more. Keep sharing such informative post keep suggesting such post.

  • 6 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels