cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Constantine
by Contributor III
  • 5060 Views
  • 1 replies
  • 1 kudos

Resolved! Can we reuse checkpoints in Spark Streaming?

I am reading data from a Kafka topic, say topic_a. I have an application, app_one which uses Spark Streaming to read data from topic_a. I have a checkpoint location, loc_a to store the checkpoint. Now, app_one has read data till offset 90.Can I creat...

  • 5060 Views
  • 1 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Hi @John Constantine​,Is not recommended to share the checkpoint with your queries. Every streaming query should have their own checkpoint. If you can to start at the offset 90 in another query, then you can define it when starting your job. You can ...

  • 1 kudos
bhargavi1
by New Contributor II
  • 2018 Views
  • 1 replies
  • 1 kudos
  • 2018 Views
  • 1 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Hi,Could you share more details on what you have tried? please provide more details.

  • 1 kudos
pbezz
by New Contributor III
  • 9777 Views
  • 5 replies
  • 9 kudos

Visualisation libraries does not work on Databricks

Why is it that certain Python visualisation libraries do not work on Databricks? I am trying to install (via pip) and work with some data visualisation libraries - they work perfectly in a normal Jupyter Notebook but not on a Databricks notebook envi...

  • 9777 Views
  • 5 replies
  • 9 kudos
Latest Reply
pbezz
New Contributor III
  • 9 kudos

No switched to using html widgets.

  • 9 kudos
4 More Replies
Manoj
by Contributor II
  • 20466 Views
  • 13 replies
  • 6 kudos

Resolved! Did any one in the community create Permanent Functions using using python script ? I tried but i am getting the below error, Please advise

Hi Team, When i am trying to register a permanant function i am getting the below error.%sqlCREATE FUNCTION simple_udf AS 'SimpleUdf'  USING JAR '/tmp/SimpleUdf.jar';%sqlselect simple_udf(2)Error Details : com.databricks.backend.common.rpc.Databricks...

  • 20466 Views
  • 13 replies
  • 6 kudos
Latest Reply
Manoj
Contributor II
  • 6 kudos

hi @Werner Stinckens​  @Jose Gonzalez​  @Hubert Dudek​ @Kaniz Fatma​ â€‹Thanks for all the help, Appreciate it. I was able to create permanent functions and use eclipse to create the runnable jar. However, Does anyone have any idea on how to deploy t...

  • 6 kudos
12 More Replies
rbarrero
by New Contributor III
  • 7340 Views
  • 9 replies
  • 7 kudos

Resolved! Error saving changes on Job Cluster

Hello all and thanks.After apply to serving a model, I go to edit corresponding Job Cluster to configure its init_script but when I try to save changes (Confirm and restart) it thrown the following error:Error: Cannot edit cluster 0503-141315-hu3wd4i...

  • 7340 Views
  • 9 replies
  • 7 kudos
Latest Reply
rbarrero
New Contributor III
  • 7 kudos

Sorry for the delay in responding. Finally a partner could fix the problem, he can edit without problems the cluster and add the init_script.Thank you!

  • 7 kudos
8 More Replies
mick042
by New Contributor III
  • 2592 Views
  • 1 replies
  • 3 kudos

Resolved! Spark Executor Rdd.Pipe call not finding file location that exists in Sparkfiles.get()

In a Databricks notebook, I need to run text files (stdin, stdout) through a function from an external library. I have used sparkContext.AddFiles({external_library_name}) to add the external library so that it is available to all executors.when I run...

  • 2592 Views
  • 1 replies
  • 3 kudos
Latest Reply
mick042
New Contributor III
  • 3 kudos

Thanks Kaniz, yes I tried that. Did not work. Falling back on init scripts now and that works.

  • 3 kudos
alefred
by New Contributor II
  • 5575 Views
  • 3 replies
  • 2 kudos

Resolved! Automation creation of Databricks Top Level Repositories

I'm looking to automate the creation of Top Level repositories in Databricks however isn't possible using cli or API if this repo is private repository(Azure DevOps Repository) because require setup the token in user setting.databricks repos create \...

  • 5575 Views
  • 3 replies
  • 2 kudos
Latest Reply
Atanu
Databricks Employee
  • 2 kudos

We have API availabe for repos https://docs.databricks.com/dev-tools/api/latest/repos.html#operation/get-repos also we are currently supporting SP. Step 1: As an admin Create a Service PrincipalUse this API SCIM API 2.0 (ServicePrincipals) | Databric...

  • 2 kudos
2 More Replies
RajibRajib_Mand
by New Contributor III
  • 5636 Views
  • 7 replies
  • 6 kudos

Unicode field separator to create unamanged table in databricks for csv file

We are getting \u318a (ㆊ) separated csv file. We want to create unmanaged table in databricks, Here is the table creation script.create table IF NOT EXISTS db_test_raw.t_data_otc_poc (`caseidt` String,`worktype` String,`doctyp` String,`brand` String,...

  • 5636 Views
  • 7 replies
  • 6 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 6 kudos

sep "\u318a"delimeter " \x318a"sep " \x318a"Try to use sep instead or/and x instead.

  • 6 kudos
6 More Replies
Robbie
by New Contributor III
  • 4152 Views
  • 2 replies
  • 4 kudos

Resolved! Why can't I create new jobs? ("You are not entitled to run this type of task...")

This morning I encountered an issue when trying to create a new job using the Workflows UI (in browser). Never had this issue before.The error message that appears is:"You are not entitled to run this type of task, please contact your Databricks admi...

Screenshot including the error message
  • 4152 Views
  • 2 replies
  • 4 kudos
Latest Reply
Robbie
New Contributor III
  • 4 kudos

@Kaniz Fatma​ @Philip Nord​, thanks!I was able to do what I needed by cloning an existing job & modifying. It's fine as a temporary fix for now.Thanks again for the response-- good to know you're aware of it & this isn't anything on my end.

  • 4 kudos
1 More Replies
SusuTheSeeker
by New Contributor III
  • 5673 Views
  • 7 replies
  • 3 kudos

Kernel switches to unknown using pyspark

I am working in jupyter hub in a notebook. I am using pyspark dataframe for analyzing text. More precisely I am doing sentimment analysis of newspaper articles. The code works until I get to some point where the kernel is busy and after approximately...

  • 5673 Views
  • 7 replies
  • 3 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 3 kudos

do you actually run the code on a distributed environment (meaning a driver and multiple workers)?If not, there is no use in using pyspark as all code will be executed locally.

  • 3 kudos
6 More Replies
auser85
by New Contributor III
  • 6977 Views
  • 1 replies
  • 3 kudos

Delta Table: Drop column failure

DBR 10.5, Spark 3.2.1```%sqlCREATE TABLE testing (    name string,    counter int    )     USING DELTA     OPTIONS (PATH "/mnt/general/testingtbl/")``````%sqlinsert into testing (name, counter) values ('a', 1)``````%sqlALTER TABLE testing SET TBLPROP...

  • 6977 Views
  • 1 replies
  • 3 kudos
Orianh
by Valued Contributor II
  • 6945 Views
  • 6 replies
  • 2 kudos

Resolved! Databrikcs job cli

Hey guys, I'm trying to create a job via databricks cli, This job is going to use a wheell file that I already upload to dbfs and exported from this package the entry point that needed for the job.In the UI I can see that the job has been created, Bu...

  • 6945 Views
  • 6 replies
  • 2 kudos
Latest Reply
Vivian_Wilfred
Databricks Employee
  • 2 kudos

Hi @orian hindi​ , adding the wheel package in the "libraries" section of json file will always try to install the whl on a cluster level that requires manage access, irrespective of job cluster or an existing interactive cluster. You cannot achieve ...

  • 2 kudos
5 More Replies
vivek_sinha
by Contributor
  • 25826 Views
  • 3 replies
  • 4 kudos

Resolved! PySpark on Jupyterhub K8s || Unable to query data || Class org.apache.hadoop.fs.s3a.S3AFileSystem not found

Pyspark Version: 2.4.5 Hive Version: 1.2 Hadoop Version: 2.7 AWS-SDK Jar: 1.7.4 Hadoop-AWS: 2.7.3When I am trying to show data I am getting Class org.apache.hadoop.fs.s3a.S3AFileSystem not found while I am passing all the information which all are re...

  • 25826 Views
  • 3 replies
  • 4 kudos
Latest Reply
vivek_sinha
Contributor
  • 4 kudos

Hi @Arvind Ravish​ Thanks for the response and now I fixed the issue.The image which I was using to launch spark executor didn't have aws jars. After doing necessary changes it started working.But still may thanks for your response.

  • 4 kudos
2 More Replies
vivek_sinha
by Contributor
  • 9986 Views
  • 3 replies
  • 4 kudos

Resolved! Getting Authentication Error while accessing Azure Blob table (wasb) URL using PySpark

I am trying to access the Azure Blob table using Pyspark but getting an Authentication Error. Here I am passing SAS token (HTTP and HTTPS enabled) but it's working only with WASBS (HTTPS) URL, not with WASB (HTTP) URL.Even I tried with Account key as...

  • 9986 Views
  • 3 replies
  • 4 kudos
Latest Reply
vivek_sinha
Contributor
  • 4 kudos

Hi @Arvind Ravish​  The issue got fixed after passing HTTP and HTTPS enabled token to spark executors.Thanks again for your help

  • 4 kudos
2 More Replies
Prabakar
by Databricks Employee
  • 3524 Views
  • 1 replies
  • 1 kudos

Non-admin users unable to create jobs from Job UI Non-admin users may be experiencing difficulties interacting with the jobs UI. This is due to a rece...

Non-admin users unable to create jobs from Job UINon-admin users may be experiencing difficulties interacting with the jobs UI. This is due to a recently discovered UI regression in the 3.73 shard release, deployed to the jobs service starting June 6...

  • 3524 Views
  • 1 replies
  • 1 kudos
Latest Reply
Prabakar
Databricks Employee
  • 1 kudos

This has been conveyed to all customers. If the email landed in your spam box then this should help you.

  • 1 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels