- 3331 Views
- 3 replies
- 22 kudos
Resolved! Package cells (scala), who uses them?
So I was wondering who uses package cells in scala?We have this library (jar) which has some useful functions we use all over the place. But that's about it. So I think we can do the same thing without a jar but with package cells.But I never hear ...
- 3331 Views
- 3 replies
- 22 kudos
- 22 kudos
Hi @Werner Stinckens​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...
- 22 kudos
- 2264 Views
- 1 replies
- 2 kudos
Is there any way to prevent non-admin users from creating new jobs?
This is specific to creating new jobs, I understand that various permissions can be set on existing jobs using job access control. This seems to suggest no, I can't find anything in the Databricks docs either.
- 2264 Views
- 1 replies
- 2 kudos
- 2 kudos
nope.Looked for that too, but it does not seem to be possible. Perhaps with Unity catalog, as there you have more permission controls.But using Unity is not an overnight decision.
- 2 kudos
- 4939 Views
- 3 replies
- 28 kudos
Resolved! Refresh Dashboard also make all related queried refresh?
Hi all,I have a quick currious. I know both query and dashboard page in Databricks SQL have refresh button to can them refresh. But one question it, when I'm in Dashboard page and click the refesh button. Does this thing also force every related quer...
- 4939 Views
- 3 replies
- 28 kudos
- 8681 Views
- 5 replies
- 2 kudos
Date field getting changed when reading from excel file to dataframe in pyspark
The date field is getting changed while reading data from source .xls file to the dataframe. In the source xl file all columns are strings but i am not sure why date column alone behaves differentlyIn Source file date is 1/24/1947.In pyspark datafram...
- 8681 Views
- 5 replies
- 2 kudos
- 2 kudos
how about using inferschema one single time to create a correct DF, then create a schema from the df-schema.something like this f.e.from pyspark.sql.types import StructType # Save schema from the original DataFrame into json: schema_json = df.s...
- 2 kudos
- 10472 Views
- 7 replies
- 8 kudos
Resolved! Is anyone else experiencing intermittent "Failure starting REPL" errors with PySpark Jobs?
I have a Multi-Task Job that is running a bunch of PySpark notebooks and about 30-60% of the time, my jobs fail with the following error:I haven't seen any consistency with this error. I've had as many as all of the tasks in the job giving this error...
- 10472 Views
- 7 replies
- 8 kudos
- 8 kudos
Hi. Did you ever got a resolution to this problem outside of rolling back to 10.4? I have recently moved some workloads over to runtime 11.3 and am experiencing intermittent "repl did not start in 30 seconds." errors.I have increased the repl timeout...
- 8 kudos
- 1165 Views
- 0 replies
- 1 kudos
- 1165 Views
- 0 replies
- 1 kudos
- 1054 Views
- 0 replies
- 3 kudos
- 1054 Views
- 0 replies
- 3 kudos
- 2081 Views
- 0 replies
- 0 kudos
How To convert the below sql server query to spark_sql. especially the isnull Replace N/A. please Help
How To convert the below query to spark sql. especially the isnull REplaceSELECT ID,ISNULL(NAME,"N/A") AS NAME,COMPANYFROM TEST
- 2081 Views
- 0 replies
- 0 kudos
- 978 Views
- 0 replies
- 0 kudos
How To convert the below query to spark sql. especially the isnull REplaceSELECTÂ ID, ISNULL(NAME,"N/A") AS NAME, ...
How To convert the below query to spark sql. especially the isnull REplaceSELECT ID, ISNULL(NAME,"N/A") AS NAME, COMPANYFROM TEST
- 978 Views
- 0 replies
- 0 kudos
- 4738 Views
- 3 replies
- 5 kudos
Getting error with spark-sftp, no such file
In a databricks cluster with Scala 2.1.1 I am trying to read a file into a spark data frame using the following code.val df = spark.read .format("com.springml.spark.sftp") .option("host", "*") .option("username", "*") .option("password", "*")...
- 4738 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi @Andreas P​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 5 kudos
- 9477 Views
- 8 replies
- 28 kudos
Resolved! Can we use Databricks or code in data bricks without learning Pyspark in depth which is used for ETL purpose and data engineering perspective.
Can we use Databricks or code in data bricks without learning Pyspark in depth which is used for ETL purpose and data engineering perspective. can someone throw some light on this. Currently learning Pyspark (basics of Pythion in handling the data) a...
- 9477 Views
- 8 replies
- 28 kudos
- 1575 Views
- 0 replies
- 2 kudos
Terraform x Databricks error INVALID_STATE subscription disabled
Hello,I just bootstrap a new Databricks EC2 on an AWS account with Terraform. Priori dependencies seems OK on my side (network, root storage, credentials configuration). I'm referring mainly to this guide and of course pages related to each Databrick...
- 1575 Views
- 0 replies
- 2 kudos
- 14346 Views
- 5 replies
- 7 kudos
Resolved! ERROR: Some streams terminated before this command could finish!
I have a databricks notebook which is to read stream from Azure Event Hub.My code does the following:1.Configure path for Eventhubs2.Read Streamdf_read_stream = (spark.readStream .format("eventhubs") .options(**conf)...
- 14346 Views
- 5 replies
- 7 kudos
- 7 kudos
I am also facing same issue , using Cluster11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12) liberary : com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.21Please help me for sameconf = {}conf["eventhubs.connectionString"] = "Endpoint=sb://xxxx.ser...
- 7 kudos
- 4436 Views
- 2 replies
- 14 kudos
Resolved! MongoDB Connection
How can we connect databricks to MongoDB?Any code reference will help
- 4436 Views
- 2 replies
- 14 kudos
- 2402 Views
- 0 replies
- 4 kudos
Optimizing a batch load process, reading with the JDBC driver
I am doing a batch load, using the JDBC driver from a database table. I am noticing in Sparkui, that there is both memory and disk spill, but only on one executor. I am also, noticing that when trying to use the JDBC parallel read, it seems to run sl...
- 2402 Views
- 0 replies
- 4 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
