- 14284 Views
- 6 replies
- 0 kudos
Resolved! How to pass outputs from a python task to a notebook task
I am trying to create a job which has 2 tasks as follows:A python task which accepts a date and an integer from the user and outputs a list of dates (say, a list of 5 dates in string format).A notebook which runs once for each of the dates from the d...
- 14284 Views
- 6 replies
- 0 kudos
- 0 kudos
Just a note that this feature, Task Values, has been generally available for a while.
- 0 kudos
- 24896 Views
- 3 replies
- 7 kudos
How to add the partition for an existing delta table
We didn't need to set partitions for our delta tables as we didn't have many performance concerns and delta lake out-of-the-box optimization worked great for us. But there is now a need to set a specific partition column for some tables to allow conc...
- 24896 Views
- 3 replies
- 7 kudos
- 1076 Views
- 0 replies
- 1 kudos
Heads up! November Community Social!  On November 17th we are hosting another Community Social - we're doing these monthly ! We want to make sure ...
Heads up! November Community Social! On November 17th we are hosting another Community Social - we're doing these monthly ! We want to make sure that we all have the chance to connect as a community often. Come network, talk data, and just get social...
- 1076 Views
- 0 replies
- 1 kudos
- 1854 Views
- 0 replies
- 8 kudos
Ask your technical questions at Databricks Office Hours October 26 - 11:00 AM - 12:00 PM PT: Register HereNovember 9 - 8:00 AM - 9:00 AM GMT: Register...
Ask your technical questions at Databricks Office HoursOctober 26 - 11:00 AM - 12:00 PM PT: Register HereNovember 9 - 8:00 AM - 9:00 AM GMT: Register Here (NEW EMEA Office Hours)Databricks Office Hours connects you directly with experts to answer all...
- 1854 Views
- 0 replies
- 8 kudos
- 2569 Views
- 2 replies
- 2 kudos
Pyspark will error while I pack source zip package without dir.
If I send the package made by zipfile on spark.submit.pyFiles which zip by this code. import zipfile, os def make_zip(source_dir, output_filename): with zipfile.ZipFile(output_filename, 'w') as zipf: pre_len = len(os.path....
- 2569 Views
- 2 replies
- 2 kudos
- 2 kudos
I checked, and your code is ok. If you set source_dir and output_filename please remember to start path with /dbfsIf you work on the community edition you can get problems with access to underlying filesystem.
- 2 kudos
- 1717 Views
- 1 replies
- 1 kudos
Checking spark performance locally
I am experimenting with spark, on my local machine. So, is there some tool/api available to check the performance of the code I write?For eg. I write:val startTime = System.nanoTime() invoicesDF .select( count("*").as("Total Number Of Inv...
- 1717 Views
- 1 replies
- 1 kudos
- 1 kudos
Please check the details about your code (task in jobs) in Spark UI.
- 1 kudos
- 6304 Views
- 1 replies
- 1 kudos
Resolved! how can I pass the df columns as a parameter
Im doing the self study and want pass df column name as a parameter.I have defined the widget column_name= dbutils.widgets.get('column_name')which is executing succefuly ( giving me a column name)then Im reading the df and do some transformation and ...
- 6304 Views
- 1 replies
- 1 kudos
- 1 kudos
df2.select([column_name]).writeORdf2.select(column_name).write
- 1 kudos
- 28970 Views
- 2 replies
- 6 kudos
Resolved! Difference between "spark.table" & "spark.read.table"?
Hi,I want to make a PySpark DataFrame from a Table. I would like to ask about the difference of the following commands:spark.read.table(TableName)&spark.table(TableName)Both return PySpark DataFrame and look similar. Thanks.
- 28970 Views
- 2 replies
- 6 kudos
- 6 kudos
Hi @Kaniz Fatma​ I selected answer from @Kedar Deshpande​ as the best answer.
- 6 kudos
- 3201 Views
- 2 replies
- 0 kudos
Faced error using Databricks SQL Connector
I installed databricks-sql-connector in Pycharm.Then i run the query below based on docs.I refer this docs.(https://docs.databricks.com/dev-tools/python-sql-connector.html)==========================================from databricks import sqlimport osw...
- 3201 Views
- 2 replies
- 0 kudos
- 0 kudos
It seems that one of your environment variables is incorrect. Please print them and compare them with the connection settings from the cluster or SQL warehouse endpoint.
- 0 kudos
- 6162 Views
- 2 replies
- 4 kudos
Resolved! How do I track databricks cluster users ?
Hi, is there a way to find out/monitor which users has used my cluster, how long and how many times in an azure databricks workspace ?
- 6162 Views
- 2 replies
- 4 kudos
- 4 kudos
Hello, You can activate Audit logs ( More specifically Cluster logs) https://learn.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/azure-diagnostic-logs It can be very helpful to track all the metrics.
- 4 kudos
- 46981 Views
- 6 replies
- 11 kudos
Resolved! how to find the size of a table in python or sql?
let's suppose there is a database db, inside that so many tables are there and , i want to get the size of tables . how to get in either sql, python, pyspark.even if i have to get one by one it's fine.
- 46981 Views
- 6 replies
- 11 kudos
- 11 kudos
@Raman Gupta​ - could you please try the below %python spark.sql("describe detail delta-table-name").select("sizeInBytes").collect()
- 11 kudos
- 9033 Views
- 1 replies
- 6 kudos
How can I simplify my data ingestion by processing the data as it arrives in cloud storage?
This post will help you simplify your data ingestion by utilizing Auto Loader, Delta Optimized Writes, Delta Write Jobs, and Delta Live Tables. Pre-Req: You are using JSON data and Delta Writes commandsStep 1: Simplify ingestion with Auto Loader Delt...
- 9033 Views
- 1 replies
- 6 kudos
- 6 kudos
This post will help you simplify your data ingestion by utilizing Auto Loader, Delta Optimized Writes, Delta Write Jobs, and Delta Live Tables.Pre-Req: You are using JSON data and Delta Writes commandsStep 1: Simplify ingestion with Auto Loader Delta...
- 6 kudos
- 2870 Views
- 0 replies
- 1 kudos
How can i save a parquet file using pandas with a data factory orchestrated notebook?
Hi guys,this is my first question, feel free to correct me if i'm doing something wrong.Anyway, i'm facing a really strange problem, i have a notebook in which i'm performing some pandas analysis, after that i save the resulting dataframe in a parque...
- 2870 Views
- 0 replies
- 1 kudos
- 1835 Views
- 0 replies
- 4 kudos
Default location for Schema/Database in Unity
Hello Bricksters,We organize the delta lake in multiple storage accounts. One storage account per data domain and one container per database. This helps us to isolate the resources and cost on the business domain level.Earlier, when a schema/database...
- 1835 Views
- 0 replies
- 4 kudos
- 1929 Views
- 1 replies
- 3 kudos
Cluster list in Microsoft.Azure.Databricks.Client fails because ClusterSource enum does not include MODELS. When you have a model serving cluster, Clu...
Cluster list in Microsoft.Azure.Databricks.Client fails because ClusterSource enum does not include MODELS.When you have a model serving cluster, ClustersApiClient.List method fails to deserialize the API response because that cluster has MODELS as C...
- 1929 Views
- 1 replies
- 3 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 466 | |
| 349 | |
| 287 |
