Databricks

Deepak_Bhutada · 10-29-2021

I have a databricks job on E2 architecture in which I want to retrieve the workspace instance name within a notebook running in a Job cluster context so that I can use it further in my use case. While the call dbutils.notebook.entry_point.getDbutils(...

Deepak_Bhutada · 08-10-2022

@tanin When you do repartition(10000), it will create 10000 tasks which will take time to launch especially if the number of cores available in your clusters is less. e.g. If you have 100 cores available in the cluster, it would have to run 100 time...

Deepak_Bhutada · 05-13-2022

@stale stokkereit You can use the below function to flatten the struct fieldimport pyspark.sql.functions as F def flatten_df(nested_df): flat_cols = [c[0] for c in nested_df.dtypes if c[1][:6] != 'struct'] nested_cols = [c[0] for c in nest...

Deepak_Bhutada · 12-20-2021

Hi @Yikun Song We found an issue with it that will be fixed in the next round of patches (to be released mid-January).As a workaround,You need to use the assembly jar like https://repo1.maven.org/maven2/org/mongodb/spark/mongo-spark-connector_2.12/3...

Deepak_Bhutada · 11-10-2021

Hi @nafri A I second what @Jose Gonzalez suggested. You can first read in spark and convert it into Delta lake, after conversion, you can run OPTIMIZE command which will coalesce all the small files into a bigger sizes. Then you can just convert it...

Deepak_Bhutada · 10-29-2021

it is not possible to get the workspace URL/Shard name from MT/E2 because we don't inject the user-visible URL, but the driver reachable one.But as a workaround to get the shardname in our job, we can write a file with our shard name in a particular ...

Databricks

User Stats

User Activity

Retrieve workspace instance name on E2 architecture (multi-tenant) in notebook running on job cluster

Re: Using .repartition(100000) causes the unit test to be extremely slow (>20 mins). Is there a way to speed it up?

Re: how to flatten non standard Json files in a dataframe

Re: Mongo Spark Connector 3.0.1 seems not working with Databricks-Connect, but works fine in Databricks Cloud

Re: How to merge small parquet files into a single parquet file?

Re: Retrieve workspace instance name on E2 architecture (multi-tenant) in notebook running on job cluster