- 4642 Views
- 1 replies
- 0 kudos
dbutils.secrets.get- NoSuchElementException: None.get
The below code executes a 'get' api method to retrieve objects from s3 and write to the data lake.The problem arises when I use dbutils.secrets.get to get the keys required to establish the connection to s3my_dataframe.rdd.foreachPartition(partition ...
- 4642 Views
- 1 replies
- 0 kudos
- 0 kudos
Howdy @Sandesh Puligundla​ - Thank you for your question. Thank you for your patience. I'd like to give this a bit longer to see how the community responds. Hang tight!
- 0 kudos
- 4378 Views
- 2 replies
- 2 kudos
Resolved! Is it possible to clone a private repository and use it in databricks Repos?
As title, I need to clone code from my private git repo, and use it in my notebook, I do something likedef cmd(command, cwd=None): process = subprocess.Popen(command.split(), stdout=subprocess.PIPE, cwd=cwd) output, error = process.communicate(...
- 4378 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Andy Huang​ , Yes, you can do it if it's accessible from Databricks. Please refer to: https://docs.databricks.com/repos.html#repos-for-git-integrationDatabricks does not support private Git servers, such as Git servers behind a VPN.
- 2 kudos
- 7703 Views
- 2 replies
- 2 kudos
Resolved! Understanding Partitions in Spark Local Mode
I have few fundamental questions in Spark3 while running a simple Spark app in my local mac machine (with 6 cores in total). Please help.local[*] runs my Spark application in local mode with all the cores present on my mac, correct? It also means tha...
- 7703 Views
- 2 replies
- 2 kudos
- 2 kudos
That is a lot of questions in one topic.Let's give it a try:[1] this all depends on the values of the concerning parameters and the program you run(think joins, unions, repartition etc)[2] spark.default.parallelism is by default the number of cores *...
- 2 kudos
- 9997 Views
- 2 replies
- 3 kudos
Resolved! current date as default in a widget while scheduling the notebook
I have a scheduled a notebook. can I keep current date as default in widget whenever the notebook run and also i need the flexibility to change the widget value to any other date based on the ad hoc run that I do.
- 9997 Views
- 2 replies
- 3 kudos
- 3 kudos
So building on the answer of Hubert:from datetime import datedate_for_widget = date.today()So if you use date_for_widget as your default value, you are there.And ofc you can fill this date_for_widget variable with anything you want.You can even fetch...
- 3 kudos
- 21892 Views
- 4 replies
- 1 kudos
Resolved! How to fix TypeError: __init__() got an unexpected keyword argument 'max_iter'?
# Create the model using sklearn (don't worry about the parameters for now): model = SGDRegressor(loss='squared_loss', verbose=0, eta0=0.0003, max_iter=3000) Train/fit the model to the train-part of the dataset: odel.fit(X_train, y_train) ERROR: Typ...
- 21892 Views
- 4 replies
- 1 kudos
- 1 kudos
Replacing max_iter with n_iter resolves the error. Thnx! It is a bit unusual to expect errors like this with this type of solution from Microsoft. As if it could not be prevented..
- 1 kudos
- 4633 Views
- 1 replies
- 2 kudos
Show all distinct values per column in dataframe Problem Statement:I want to see all the distinct values per column for my entire table, but a SQL que...
Show all distinct values per column in dataframeProblem Statement:I want to see all the distinct values per column for my entire table, but a SQL query with a collect_set() on every column is not dynamic and too long to write.Use this code to show th...
- 4633 Views
- 1 replies
- 2 kudos
- 2 kudos
@Artem Yevtushenko​ - This is great! Thank you for sharing!
- 2 kudos
- 14750 Views
- 8 replies
- 5 kudos
Resolved! error creating tables using UI
Hi, i try to create a table using UI, but i keep getting the error "error creating table <table name> create a cluster first" even when i have a cluster alread running. what is the problem?
- 14750 Views
- 8 replies
- 5 kudos
- 5 kudos
Be sure that cluster is selected (arrow in database) and at least there is Default database.
- 5 kudos
- 33225 Views
- 11 replies
- 10 kudos
Resolved! Read JSON files from the s3 bucket
Hello guys, I'm trying to read JSON files from the s3 bucket. but no matter what I try I get Query returned no result or if I don't specify the schema I get unable to infer a schema.I tried to mount the s3 bucket, still not works.here is some code th...
- 33225 Views
- 11 replies
- 10 kudos
- 10 kudos
Please refer to the doc that helps you to read JSON. If you are getting this error the problem should be with the JSON schema. Please validate it.As a test, create a simple JSON file (you can get it on the internet), upload it to your S3 bucket, and ...
- 10 kudos
- 7369 Views
- 7 replies
- 0 kudos
data from 10 BLOB containers and multiple hierarchical folders(every day and every hour folders) in each container to Delta lake table in parquet format - Incremental loading for latest data only insert no updates
I am able to load data for single container by hard coding, but not able to load from multiple containers. I used for loop, but data frame is loading only last container's last folder record only.Here one more issue is I have to flatten data, when I ...
- 7369 Views
- 7 replies
- 0 kudos
- 0 kudos
for sure function (def) should be declared outside loop, move it after importing libraries,logic is a bit complicated you need to debug it using display(Flatten_df2) (or .show()) and validating json after each iteration (using break or sleep etc.)
- 0 kudos
- 3426 Views
- 1 replies
- 6 kudos
Resolved! Are there data types that are not good in Delta format? Does Delta handle images, audio, and video?
- 3426 Views
- 1 replies
- 6 kudos
- 6 kudos
Hi as it is transaction tables (there are history commits and snapshot). I would not store there images or videos as it can be saved few times and you will have high storage costs, it can also be slow when data is big.I would definitely store images,...
- 6 kudos
- 5126 Views
- 4 replies
- 10 kudos
Resolved! How to make sparklyr extension work with Databricks runtime?
Hello. I'm the current maintainer of sparklyr (a R interface for Apache Spark) and a few sparklyr extensions such as sparklyr.flint.Sparklyr was fortunate to receive some contribution from Databricks folks, which enabled R users to run `spark_connect...
- 5126 Views
- 4 replies
- 10 kudos
- 10 kudos
Yes, as Sebastian said. Also, it would be good to know what the error is here. One possible explanation is that the JARs are not copied to the executor nodes. This would be solved by Sebasitian's suggestion.
- 10 kudos
- 1595 Views
- 1 replies
- 0 kudos
- 1595 Views
- 1 replies
- 0 kudos
- 0 kudos
@Rajeswari Gummadi​ - Is this a duplicate of your other thread? I don't see any content and I want to make sure all your questions are answered.
- 0 kudos
- 4606 Views
- 3 replies
- 17 kudos
Resolved! Azure: Permanently purge cluster logs
Is there any way to purge logs via API instead of clicking daily that option:
- 4606 Views
- 3 replies
- 17 kudos
- 7726 Views
- 2 replies
- 1 kudos
AssertionError: assertion failed: Unable to delete the record but I am able to select it though
Is there any reason this command works well:%sql SELECT * FROM datanase.table WHERE salary > 1000returning 2 rows, while the below:%sql delete FROM datanase.table WHERE salary > 1000ErrorError in SQL statement: AssertionError: assertion failed:...
- 7726 Views
- 2 replies
- 1 kudos
- 1 kudos
DELETE FROM (and similarly UPDAT. aren't supported on the Parquet files - right now on Databricks, it's supported for Delta format. You can convert your parquet files into delta using CONVERT TO DELTA, and then this command will work for you.
- 1 kudos
- 15289 Views
- 4 replies
- 4 kudos
Resolved! Unable to save Spark Dataframe to driver node's local file system as CSV file
Running Azure Databricks Enterprise DBR 8.3 ML running on a single node, with Python notebook. I have 2 small Spark dataframes that I am able source via credential passthrough reading from ADLSgen2 via `abfss://` method and display the full content ...
- 15289 Views
- 4 replies
- 4 kudos
- 4 kudos
Modern Spark operates by a design choice to separate storage and compute. So saving a csv to the river's local disk doesn't make sense for a few reasons:the worker nodes don't have access to the driver's disk. They would need to send the data over to...
- 4 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
